TK, If it helps, I don't use any reporting stuff. I'm strictly using WMR300 with rapid fire and archive simultaneously. No other features.
Unfortunately, I've restarted weewxd several times today making the attempt to debug it, so I can't share the latest memory usage snapshot. But as a data point my watchdog has to reboot the pi roughly every 3 days. So now that I've restarted weewx I'll see if it goes 6 days instead of the usual 3. Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad) > On Aug 11, 2019, at 7:22 PM, Thomas Keffer <tkef...@gmail.com> wrote: > > > While the tools work well in smaller projects where you can identify and > track every object, in a larger, real-time, project like weewx, there is just > too much going on to make sense of the huge memory trees the tools produce. > > They also suffer from a Heisenberg uncertainty principle: the very act of > measuring consumes memory, which you then chase, looking for your leak. > > So, I have taken to a patient, binary approach. I shut down half the program, > say the reporting engine, and see if that makes a difference. If not, I shut > down the other half. Keep dividing by half and eventually you'll locate the > problem. With most leaks, you can only do one iteration every day or two, so > it can take weeks to isolate the problem. With a large leak, like yours, it > can be done in a matter of hours. > > Patience is the only thing I've found that works well! > > One thing worth noting: Python uses garbage collection, so it doesn't > actually "leak" memory. It may struggle to reclaim memory that has cyclic > references, taking a long time to reclaim it, but eventually it will (except > if a __del__ method is defined and when using a Python version before 3.4. > Weewx never defines __del__, so it is not subject to this limitation.). > > Most of the time the problem is in an underlying "C" routine, specifically > drivers, occasionally graphic utilities. > > -tk > >> On Sun, Aug 11, 2019 at 3:51 PM Leon Shaner <l...@isylum.org> wrote: >> Hey, TK, >> >> Can you please toss me a bone on how to instrument weewx for memory >> profiling? >> I tried python3-memory-profiler but couldn't make it work. It seems to >> block the normal importing of needed python modules (like weewx.engine). >> Maybe it is because weewx uses precompiled modules? >> >> Other than that I found a thread in the weewx google group where you >> suggested pympler, but I can't seem to find that on Raspbian, and you had a >> specially modified engine.py anyway. :-/ >> >> Regards, >> \Leon >> -- >> Leon Shaner :: Dearborn, Michigan (iPad) >> >>>> On Aug 11, 2019, at 4:06 PM, Leon Shaner <l...@isylum.org> wrote: >>>> >>> Hey, TK. WMR300. I'll see if I can track it down. >>> >>> Regards, >>> \Leon >>> -- >>> Leon Shaner :: Dearborn, Michigan (iPad) >>> >>>>> On Aug 11, 2019, at 12:45 PM, Thomas Keffer <tkef...@gmail.com> wrote: >>>>> >>>> >>>> If memory increases every couple of seconds, which suggests that the leak >>>> is in the part of device driver that deals with LOOP packets. But, you're >>>> on an WMR300, right? As I recall, their LOOP packets are more like 10-20 >>>> seconds apart --- too long to correlate with what you're seeing. Or, are >>>> you on something like a Vantage with a shorter LOOP packet interval? >>>> >>>> I've noticed memory leaks in other instances, such as my own site >>>> http://www.threefools.org/weewx. In my case, I spent a couple of weeks >>>> trying to chase it down, and concluded the problem was somewhere in the >>>> sqlite database driver, but couldn't put my finger on it conclusively. >>>> >>>> Then, magically, Debian did an update and the problem (nearly) went away. >>>> Memory leak dropped from 5 MB/day, to almost nothing (maybe a megabyte or >>>> two over a week). Again, it suggests the problem was (is) in a device >>>> driver somewhere. >>>> >>>> -tk >>>> >>>>> On Sun, Aug 11, 2019 at 8:55 AM Leon Shaner <l...@isylum.org> wrote: >>>>> TK, you may want to know... >>>>> There definitely is a memory leak in weewxd. >>>>> It grows by 1 KiB about every 2 seconds. >>>>> >>>>> I haven't yet correlated whether weewxd can grow to a point where other >>>>> things start dying, and whether that has any impact on the USB stack. >>>>> >>>>> What tipped me off was I just got a warning from sar/sa1 stating it >>>>> couldn't do it's job due to "resource unavailable," so I went and had a >>>>> peek. >>>>> >>>>> This RPI is a Zero W(H), so it has only 1 GB of RAM. >>>>> It's full enough on memory that it's using some swap. >>>>> >>>>> Here is just one snapshot. That RES (RZ) value for weewxd goes up by one >>>>> digit every 2 seconds. >>>>> I gleaned that is 1 KiB, but I am not 100% certain. It might only be 1 >>>>> byte every 2 seconds -- whatever the actual unit it goes up by one every >>>>> 2 seconds. I've never seen it come down. >>>>> >>>>> I may enhance my watchdog to check the weewxd memory size and restart it >>>>> after a limit, then see if that has any positive impact on avoiding the >>>>> USB stack going south. Yeah, I guess I'll work on that this week. I'll >>>>> put some logging in so anyone who wants to keep an eye on it can know >>>>> whether their setup is memory leaking also. =D >>>>> >>>>> I have the WMR300 driver, in case the leak could be specific to that >>>>> driver. :-/ >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "weewx-user" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to weewx-user+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/weewx-user/D7820F76-CC3A-4CC2-B359-4AF967A32638%40isylum.org. >>>>> >>>>> >>>>> Regards, >>>>> \Leon >>>>> -- >>>>> Leon Shaner :: Dearborn, Michigan (iPad) >>>>> >>>>> > On Aug 10, 2019, at 9:01 AM, Leon Shaner <l...@isylum.org> wrote: >>>>> > >>>>> > It isn't the weewx version, it's the pi kernel / usb module version. >>>>> > The usb stack hangs. I wrote a watchdog to detect it and reboot the >>>>> > pi, because so far there has been no fix to the issue. I am not able >>>>> > to correlate this to a memory leak. What are you seeing re: memory? >>>>> > >>>>> > Regards, >>>>> > \Leon >>>>> > -- >>>>> > Leon Shaner :: Dearborn, Michigan (iPhone) >>>>> > >>>>> >> On Aug 10, 2019, at 4:29 AM, Gordie Stirling <stir...@gmail.com> wrote: >>>>> >> >>>>> >> Hi all, >>>>> >> >>>>> >> I have been runnin Weewx on a Raspberry Pi (model a) for a couple of >>>>> >> years with a Maplin NG96Y, without any issues. >>>>> >> >>>>> >> I recently upgraded to a Pi3, running Stetch, and Lighttpd. the old >>>>> >> Pi was starting to struggle and sat at 100% cpu for most of the time. >>>>> >> >>>>> >> All worked well for a month or so, the the station locked up. After a >>>>> >> reset(batteries out, usb disconnected) it works for an hour or so, >>>>> >> then locks the connection again. the base station still continues to >>>>> >> function. >>>>> >> >>>>> >> Error in syslog is Aug 10 09:17:28 raspberrypi weewx[10377]: fousb: >>>>> >> get_records failed: [Errno 110] Operation timed out. >>>>> >> >>>>> >> so far, other than rebooting, i have used Easy Weather to clear tge >>>>> >> memory, but alas had no effect, 85 mins later, timeouts start again. >>>>> >> >>>>> >> I am thunking of downgrading the Pi to Jessie, and using reverting to >>>>> >> Weewx 3.9.1, but wondered if anyone has had this problem and managed >>>>> >> to fix rather than rebuild. >>>>> >> >>>>> >> TIA >>>>> >> >>>>> >> G >>>>> >> >>>>> >> -- >>>>> >> You received this message because you are subscribed to the Google >>>>> >> Groups "weewx-user" group. >>>>> >> To unsubscribe from this group and stop receiving emails from it, send >>>>> >> an email to weewx-user+unsubscr...@googlegroups.com. >>>>> >> To view this discussion on the web visit >>>>> >> https://groups.google.com/d/msgid/weewx-user/afbf7556-425f-4eca-8088-caaf96c06805%40googlegroups.com. >>>>> > >>>>> > -- >>>>> > You received this message because you are subscribed to the Google >>>>> > Groups "weewx-user" group. >>>>> > To unsubscribe from this group and stop receiving emails from it, send >>>>> > an email to weewx-user+unsubscr...@googlegroups.com. >>>>> > To view this discussion on the web visit >>>>> > https://groups.google.com/d/msgid/weewx-user/315802A3-3CD7-4206-B22B-77A377B0F5A7%40isylum.org. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "weewx-user" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to weewx-user+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/weewx-user/D7820F76-CC3A-4CC2-B359-4AF967A32638%40isylum.org. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to weewx-user+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/CAPq0zECp12D-2O0S10NOGFOZQFUSMrjPSdYgDwH3nf3tQL5sHw%40mail.gmail.com. >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to weewx-user+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/F83E049D-D09A-432A-8880-626C38728743%40isylum.org. > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/CAPq0zECadyvwsVv3xvHTbJx5RQDoK6mCA11_W1utok8DYbpHuw%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/45A0751F-377D-431C-89FF-F41E1D50AF09%40isylum.org.
