Many, many thanks! Before I wrote the service to investigate the problem, I wrote a standalone program to eliminate all WeeWX components. It did NOT exhibit the problem.
Problem only seems to show up under WeeWX/Belchertown *and* if the include file changes / is recreated. Anyway, I really appreciate you looking into my problem. I’m actively working on a skin-independent service to read many RSS/Atom feeds and will utilize your suggestions above. Regards, Garry Lockyer C: +1.250.689.0686 E: ga...@lockyer.ca > On Jan 9, 2021, at 12:39, bell...@gmail.com <bellri...@gmail.com> wrote: > > Garry, > I took your WxFeedsMemory.py and made some modifications. > First, I changed the logging to simple print calls. this was just a > convenience for me to easily see what was going on. > Second, I stole some code from Vince’s mem extension, > https://github.com/vinceskahan/vds-weewx-v3-mem-extension. I did this because > I am used to the data it returns. (Note, I highly recommend this when > debugging memory.) > Third, I added code so I could call your extension repeatedly in a loop. > > When I called new_archive_record 100 times there was a slight increase in > memory, but nothing alarming. But it was slow, approximately 2 minutes a call. > > So, I commented out the flush and fsync calls and called new_archive_record > 10,000 times. The slight increase seemed to stop around 6,000 calls. > > I’m wondering if the amount of time it takes to run is causing problems when > run as a service. I plan to uncomment the code and run it as an extension. > I’ll let you know what I see. > rich > > > On Thursday, 7 January 2021 at 13:00:40 UTC-5 garrya...@gmail.com wrote: >> After many hours of troubleshooting and testing, I think I have an idea >> what's happening. >> >> Background: I want to use weewx with the Belchertown skin and my extension >> that reads numerous RSS/Atom feeds on a Pi Zero, with either a Davis Pro V2 >> or Weatherflow weather station. I will eventually have four weather >> stations in my area. I thought I had things working reasonably well so >> deployed one station. It stopped working after about 3 days. I can't >> access that system remotely so I built a very similar system (the next one >> to deploy) and it falso ailed due to lack of memory. Weewx memory usage on >> my development system also increases over time - it can grow to over 1 GB in >> a few hours! >> >> My extension was an extension to Belchetown based on the METAR extension. >> The extension created Belchertown index hook include files each archive >> cycle. While researching this problem I re-read the weewx customization >> guide and noted that extensions should NOT be dependent on other extensions >> so I decided to re-write my extension as a service (which looks like a much >> cleaner solution) without any dependencies on Belchertown (other than >> include file names). All I've done so far is create a service >> (WxFeedsMemoryTest.py) to test weewx/Belchertown memory consumption. >> >> The Problem: weewx/Bechertown memory usage increases over time. It starts >> out at about 45 MB and grows at about 3MB per archive period/cycle (using my >> test case). A 512 MB Pi will exhaust memory within a few days. >> >> It appears that the problem is associated with the creation of the >> Belchertown include files while weewx/Belchertown is running: >> >> - if the include file is 'static' as in not (re)created while >> weewx/Belchertown is running, memory usage is static - it does not grow >> beyond about 50 MB. >> >> - if the include file is 'dynamic' as in (re)created while weewx/Belchetown >> is running, memory usage increases. >> - if the include file is created once, and becomes 'static', memory usage >> increases and then stabilizes. >> - if the include file is recreated continuously (such as on each archive >> cycle), memory usage increases each cycle. >> >> It does not appear to matter if the include file is created directly, or >> created as a temporary file and then copied or renamed. >> >> The attached service (WxFeedsMemoryTest.py) can be used to demonstrate the >> problem. Please see installation and use instructions within the >> WxFeedsMemoryTest.py. >> >> I'm going to continue to work on moving my extension from "an extension to >> an extension" to a service in the hope that this memory problem can be >> resolved. >> >> With apologies in advance if I'm doing something to cause the problem, >> please review, advise and let me know what I can do to avoid the problem. >> >> Regards, >> >> Garry >> On Thursday, December 31, 2020 at 7:05:15 PM UTC-8 garrya...@gmail.com wrote: >>> Got MemoryError after about 9 hours after restart. Have removed cmon by >>> commenting out any mention of cmon in weewx.conf and restarted. >>> >>> Regards, >>> >>> Garry Lockyer >>> Former DEC Product Support Engineer :^) >>> Kepner-Tregoe Trained :^)) >>> C: +1.250.689.0686 >>> E: ga...@lockyer.ca >>> >>> >>>> On Dec 31, 2020, at 11:44, vince <vince...@gmail.com> wrote: >>> >>>> On Thursday, December 31, 2020 at 11:39:39 AM UTC-8 garrya...@gmail.com >>>> wrote: >>>>> Re: editing the Belchertown skin, nope haven’t touched it, other than >>>>> interfacing with it via the include files (as generated by my >>>>> BelchertownWxFeeds extension). When all the endpoints (for testing) are >>>>> enabled index.html is about 1.8MB, so perhaps that’s causing the problem. >>>>> I can easily reduce / eliminate endpoints and prefer to do that before >>>>> eliminating the Belchertown skin. >>>>> >>>> >>>> There it is. You touched it :-) >>>> >>>> Usual debugging rules apply. Reset it to a baseline unmodified config. >>>> Add in changes one-by-one. If it goes sideways, revert to the last known >>>> good and reverify that it stays good. >>>> >>> >>>> -- >>> >>>> You received this message because you are subscribed to the Google Groups >>>> "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to weewx-user+...@googlegroups.com. >>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/57a5bfc7-d0b4-4419-b178-6342564642edn%40googlegroups.com. > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/d5e41e91-5dab-41bc-bc34-682826ba4753n%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/B14DEE29-3141-4DBF-95A9-231E17999638%40gmail.com.
