Daily Energy steps up massively every time consumption resumes from 0

[kineticscreen]

After having stabilised the instantaneous consumption data errors I was getting by adding the internal filter: as well as ensuring no feedback from flashing LEDs, this has revealed a different issue, where the Daily Energy increases by huge numbers every time the unit starts to detect consumption again, after a period of 0 flashes due to adequate solar generation:

As you can see, this not the result of an incorrect reading of the Consumption - only momentary modest pulling from the grid is accompanied by massive instantaneous jumps in the daily total.

Has anyone experienced this? My only thought it is somehow connected to the 4-5 minute lag the unit has to changing power consumption to 0 once the flashing has stopped. And somehow when it kicks back in, it's assuming a value across the time that should be 0?

[kineticscreen]

So further to this, it seems like the size of the step up is connected to how long the sensors has been 'off' or detecting zero, because this is the ridiculous jump it did today, a day of continuous solar production, and then when the Glow kicked in at the end of the day, it instantaneously added nearly 18kWh to the total.

[github-actions]

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

[kineticscreen]

This issue continues, and renders the Glow useless for daily power consumption, and only useful for instantaneous power monitoring. This issue shouldn't be closed off, even if there is no solution, because people need to be able to find that it as a serious limitation of the platform.

[klaasnicolaas]

I think the best solution at this point is to put a note somewhere in the documentation that it is best not to use glow if you also have solar panels. What you also can try is using a different method: left or trapezoid.

./Klaas

[kineticscreen]

What you also can try is using a different method: left or trapezoid.

Happy to try this out - is there a way I can make this change onboard the Glow itself, rather than making a Template sensor in HA - would be good to still get the advantage of the 'always recording' glow, even during a system reboot, etc.

[hawksj]

I have noticed this in testing as the test pulses I've been using are not well timed and I will wait long periods before testing it again. I think Klaas means you can change the method of the total_daily_energy sensor by adding this to the adopted devices ESPHome YAML:

sensor:
  - id: !extend sensor_total_daily_energy
    method: left

Petro on the HA Community Forum has a great explanation of this here, where he also suggests that trapezoid is not good if you have large periods of the same value (or 0, in the case of solar):
https://community.home-assistant.io/t/calculating-watts-kw-and-kwh/173250/3

There is also this post which says that for square waveforms, the left method should be used.
https://community.home-assistant.io/t/integral-trapezoidal-rule-incorrect-data-for-power-energy/771985/4

This is totally untested and only based on some research and assumptions on what Klaas meant because I have also noticed this issue. I will report back if/when I get a chance to test.

[kineticscreen]

Can confirm that changing to left does indeed dramatically reduce the problem, though you can still see the drift against a clamp meter.

[hawksj]

I think that is a limitation of the pulse meter method in general. The pulse meter only knows the data that is there (the pulses), it makes assumptions about what the value was in between polling samples because the Riemann sum is an approximation. The CT clamp has a much higher polling rate in comparison. Have a look at the Wikipedia page for graphs that visualise the methods: https://en.wikipedia.org/wiki/Riemann_sum

Despite that, I think maybe you'd got some false signals in your data, or noise? I think that even if the approximations of energy consumption are wrong, the total at the end of the day should be correct because for every x kWh consumed, there must have been x * pulse_rate pulses sent. The fact it doesn't add up with your CT Clamp at the end of the day, I think, means it must have counted some pulses that didn't happen.

Disclaimer, it's a Friday morning before I finish work for Christmas so take the above with a grain of salt.

[kineticscreen]

I think you're right, though I imagine the "drift" that is still happening between the glow and the CT clamp is the same "stepping up" when the glow goes to 0, however it is now just happening at the 'left' moment, so it isn't as dramatic a result.

There's also the chance that the clamp is slightly under-measuring the actual current, but this is a notable improvement, as in the past it was out by orders of magnitude - multiple kWh a day.

[hawksj]

The graph does suggest the "stepping up" is the cause, based on the CT clamp being flat. I've just checked the total daily energy component page and it says it uses the integral. I'll admit complex math is not one of my strengths but I can't understand why it uses an integral. Counting total daily energy with a pulse meter is as simple as dividing the number of total pulses received in a 24 hour period by the pulse rate.

@klaasnicolaas am I missing something critical here? It seems to be that doing the above and simply counting every pulse as 1/pulse_rate kWh would give more accurate numbers than relying on integrals and assumptions. The total of the pulse meter, for instance, reset every 24 hours, would be more accurate at counting daily energy usage than the total_daily_energy sensor which is intended to take a power input in watts and estimate the daily usage.

A template sensor that calculates the difference between the value of total_sensor_energy at 00:00 and now would be pretty much perfectly accurate, assuming a perfect input signal, no? If this number was updated frequently enough you'd have a pretty nice smooth graph similar to Kinetic's CT clamp graph, rather than one with big steps and jumps.