Gas Meter Monitoring

To avoid swapping tabs, I'll copy my most recent update from that thread:

I have not yet finished this project, but I wanted to post a quick update on my progress. Hopefully this will make it more clear what I'm doing and how I'm doing it.


Here is a photo of the device sitting on top of the gas meter, powered by a portable usb charger:



And here is the magnetometer dangling down the back side of the meter:



You are looking at $10 worth of parts.

This is obviously a temporary setup for testing purposes, and placement of the sensor is not ideal, but here is a sample of the data I'm getting anyway:




Absolutely beautiful! Just hanging out back there the magnetometer is picking up useful differences in field strength as the meter spins.

Looking at the chart:

4:37 Placed the device on the meter
4:38 Lit big burner on stove
4:40 Lit *all* the burners on the stove
4:42 Turned off stove

After 4:42 is the slow background gas consumption of the various pilot lights in the house (plus any small leakage I guess), which by the way looks to be 2 cubic foot burned per hour or nearly 1,500 per month. If my math is correct it costs me nearly $20/month just to keep pilots lit! That sounds like a handy figure to know when considering switching from gas to electric (or going tankless).

According to my eyeball calibration watching the chart and the dial, 8 high-low cycles on my chart equals 1 cubic foot of gas. Next step is to work on the code to count cycles instead of just reporting the raw data. Once I'm counting cycles it's trivial to convert to realtime/hourly/daily consumption rates, perhaps even reported in dollars. Then of course I need to put it in a watertight enclosure and mount it to the side of the house with a small solar panel.

I have other projects in the works of similarly low cost to gather data on water heater temps. That in conjunction with the data I already have from my smart thermostats will allow me to set up automations which can trigger excess gas consumption when neither the water heaters nor the furnaces are running. As of now this would trigger whenever I run the stove or the gas fireplace, but I can also develop devices to detect the state of these appliances as well.


Maybe all of this is mostly pointless, but for me it was the missing piece in monitoring all my utility consumption. Electricity and water have commercial options that are reasonably priced, but I couldn't find anything for gas so I built it. I think at least a few people in the world would find it useful, maybe restaurants or other small businesses that might like to keep an eye on their gas consumption.

quote:

---

What hardware are you actually running HA on?

---

Presently it's running on a Pi, but I am currently in the planning and purchasing stages (as I have been for 2 years now it seems) of building out a "homelab" cluster. It will be a proxmox+ceph "hyperconverged" cluster on used mini pc's from ebay. Presently in addition to HA on the Pi I have a single server which runs Plex (plus associated media apps), UniFi controller, security cam NVR, and misc other services. This single point of failure brings all my shite down, and I've already replaced the power supply once.

So my new cluster will distribute both compute and storage resources across machines while behaving more like a single server. All of my services will be in either a VM or a container, so proxmox high availability will automatically move a service to a new machine if one goes down or I take it out of service. Similarly with storage, ceph handles redundancy behind the scenes so it will shuffle data around to maintain a minimum number of copies as machines are added or subtracted. I think performance will be less-than-stellar since I won't have 10gbit networking dedicated to ceph, but I'm fine with slow transfers for now. Compared to a typical NAS box, I think the flexibility of being able to add random hardware and mismatched storage at will outweighs the performance drawback for me.

quote:

---

in the DEV lab at the office is limited to 1G but still performs pretty well

---

Running ceph on that?

quote:

---

@kork I'd use NodeRed to get the rate out of that sine wave. I know it has the capabilities. Not sure HA has a one-shot counter which is what you'd need to count the cycles.

---

I'm going to do the counting on the ESP. Right now it's just sending raw sensor data to HA, which is kind of a lot of data and HA doesn't really need it. I can have ESPHome watch the signal and just increment a counter at the threshold that I set and it'll just report minute-by-minute consumption to HA. A lot less data hitting the HA database, and a lot less compute work on HA's end.

I would consider a single, more powerful machine then trying to cluster multiple machines over a 1 gig link. Clustering in Proxmox can also get a little messy, especially after outages. It's why I've moved to more of a singular server for virtualization (I've got my production server and test server).

Have you tried using a reed switch and just counting pulses on a digital port?

quote:

---

in the DEV lab at the office is limited to 1G but still performs pretty well
Running ceph on that?

---

Everything is running over NFS to ZFS (TrueNAS), haven't had time to play with ceph yet, but it's on my short list.

So this thread got derailed a bit, so bump to get back on track.

As with most of my projects, this one went on the back burner (heh) for a bit, but I recently got back on it and have started pumping my natgas consumption data into HA.

I ended up scrapping the battery+solar idea and ran an ethernet cable to power the ESP via PoE. I also used ethernet cable to connect the magnetometer to the ESP so that the enclosure could be mounted to the house and only the cable with tiny sensor attached is actually on the meter, so it's pretty discreet. I weatherproofed and mounted the sensor to the back of the meter with FlexTape.

First a few sample charts of the data I'm getting:


A couple hours' worth of raw magnetic field strength data. The only consumption here is a few pilot lights plus whatever leakage I have. This is just over 16 cycles (rotations of disk inside meter) per hour, which equates to 2 cubic feet per hour.


Here is the above data converted to flow rate. You can see it's centered on 2 cuft/hr, though the "steps" are quite apparent. I'll explain the cause below.


And now the above data as cumulative consumption.




Zooming in a bit, here is a section from last night when a water heater kicked on. The slow cycles are my "background" 2 cuft/hr consumption, and it is obvious where the water heater starts and stops consuming ~30 cuft/hr.


The above data as flow rate over time. Again notice the "steps".


And the cumulative consumption over that period.


And Home Assistant's helpful Energy dashboard chart.





So now I'll get into the weeds of the issues I ran across and how I solved them.

It took me a few hours of fiddling with code to make it work reasonably well. Although it's easy to see the pattern visually and count the peaks and valleys over time, doing that in code is an entirely different matter. The data is very noisy, which is usually resolved with some type of moving average or other method of smoothing out the data. The problem with this is as the frequency of the cycles increases, the magnitude of the peaks and valleys decreases significantly, to the point that if the flow rate is high enough you end up with essentially a flat line. That can be mostly resolved with a faster sensor poll rate, but that makes the data even noisier. I went in circles for a while trying different parameters for poll rates and EMA smoothing and such trying to find something that works well for very slow and very fast flow rates, but eventually I gave up and decided to just use the raw noisy data.

I think scrapping any data smoothing in a futile attempt at making consistent peak and valley values worked out for the best. It allowed me to think about it differently, and I arrived at a method that not only counts cycles reliably but is also self-calibrating. The self-calibration is quite nice because I can relocate the sensor (which results in the field strength cycling between a different range of values) without changing any hard-coded values. I won't post any code just yet because it is far from production-ready, but I'll explain the technique.

At boot, we start collecting sensor values. The first two values become the high and low values of the range, and the midpoint is calculated. These values, perhaps obviously, will be pretty similar given that they were measured back-to-back, but that doesn't matter. The third value can be in any of 4 regions of our chart: above our old max value, below max but above the midline, below midline but above our old min, or below our old min. If it's above the max, we set the value as our new max, and critically for my solution at the same time we adjust the min to halfway between the old minimum and the old midpoint, then calculate the new midpoint value. Likewise if we set a new minimum for the range, we adjust the old max value down and recalculate the midpoint.

So when the value touches a min or max and sets a new limit to the range, I flip my "dial position" to either positive or negative. This is how I'm counting cycles/pulses. It is counted only the first time a limit is touched, and the dial position state prevents repeated increases in the min or max from continually moving the opposite limit. I'm not sure if that's a clear explanation.

Anyway, maybe you can see in the sine wave that the peaks are a little wider/fatter than the valleys, which I think is due to the magnetic part of the disk being closer/further from the sensor at those points. And that is the cause of the "stepped" flow rate data. My "dial position" flips quicker in the valleys than it does in the peaks, resulting in variance in the flow rate calculation. I could solve this by only counting full cycles as opposed to half cycles, but then I would only get flow rate updates every ~8ish minutes vs. the ~4 minutes I get them now. See the flatline in flow rate after the water heater cut off? That would be twice as long if I were counting full cycles only.

I could go a step further and estimate the actual angle of the disk, which would give me extremely fine-grained data and to-the-second flow rate updates, but as it is I'm counting each 0.06 cubic feet of gas which I think is precise enough.




So that's about it for now. As I calculated before, it looks like my pilot lights cost about $20/month to keep burning, which is insane to me. I have two water heaters which I believe burn about 1/2 cuft/hr each, and I guess my furnace burns about 1 cuft/hr keeping the pilot lit. Like $75 worth of gas burning a pilot light on the furnace from March to November for no reason whatsoever. Crazy.

Once concern I have is whether I'm polling fast enough to keep up with the flow rate that my generator will pull. It works for the 30 cuft/hr of the water heater, but I believe the generator will need about 200 cuft/hr. I might only get a handful of data points per wave cycle. Yep I think I'll need to bump it up for that.