Just a followup - I took belchertown out of the picture and used my minimal demo-skin and added 'one line' that #include(d) a file generated by Garry's service, and it 'is' leaking memory albeit less quickly than the belchertown example.....
I'll let it run overnight and we'll see what it looks like after 12 more hours running. The VM is set to only 1GB RAM but I'm hoping if it runs out it'll just go into swap and stay up. OS is debian 10 under Vagrant/VirtualBox with 4.3.0 installed using python3 and setup.py Some mem.sdb output (date/time, units, interval, mem_size, mem_rss, mem_share)... 1611109216 16 5 141.0625 71.734375 12.2734375 1611109516 16 5 142.5625 75.65234375 12.3359375 1611109816 16 5 144.0625 79.65625 12.3359375 1611110116 16 5 145.5625 84.5234375 12.3359375 1611110416 16 5 147.24609375 88.0859375 12.3359375 1611110716 16 5 148.49609375 91.43359375 12.3359375 1611111016 16 5 149.99609375 95.421875 12.3359375 1611111316 16 5 151.49609375 100.3359375 12.3359375 1611111616 16 5 152.99609375 103.8359375 12.3359375 1611111916 16 5 154.49609375 107.51171875 12.3359375 1611112216 16 5 155.99609375 111.5 12.3359375 1611112516 16 5 157.24609375 116.0859375 12.3359375 1611112816 16 5 158.74609375 119.5859375 12.3359375 1611113116 16 5 160.24609375 123.24609375 12.3359375 1611113416 16 5 161.74609375 127.234375 12.3359375 1611113716 16 5 163.24609375 132.11328125 12.3359375 On Tuesday, January 19, 2021 at 5:21:31 PM UTC-8 bell...@gmail.com wrote: > Ok, my standalone testcase was wrong. I think I have a valid one here, > https://github.com/bellrichm/experiments > It just calls Cheetah.Template.Template in a loop. If the include file > modified attribute does not change, all seems stable. If it changes, memory > usage seems to grow. > rich > > On Tuesday, 19 January 2021 at 19:16:46 UTC-5 tke...@gmail.com wrote: > >> I'm not quite following this discussion. >> >> With Cheetah, you can compile the template and save the results. I >> believe this is what monitors the changed status of #include files. But, >> that's not the way WeeWX uses templates. It compiles, evaluates, then >> throws the results away. >> >> If you use the "raw" directive in an #include, then the file will not be >> parsed. So, how can any $ directives work? >> >> Or, does Belchertown work differently? >> >> -tk >> >> On Tue, Jan 19, 2021 at 3:55 PM bell...@gmail.com <bell...@gmail.com> >> wrote: >> >>> I agree that something does not seem quite right with Cheetah’s >>> ‘monitoring’ of include files. I took a different approach. >>> First, I wrote a simple WeeWX service that would perform the ‘touch’ >>> command against the large file and a simple template that included the >>> large file. When the file modified attribute changed, I observed the memory >>> growth, otherwise all was stable. Also, when the file was not ‘touched’ the >>> Cheetah processing was noticeably faster. >>> Next I wrote a standalone program to run a loop that ran the touch >>> command and invoke Cheetah. I ran it 3 times. Each time around loop 90, >>> the memory growth stopped and the processing was fast as though Cheetah was >>> using cached data. >>> I am back to running under WeeWX to see if it will level off like it >>> does standalone. I am also wading through the Cheetah code and >>> documentation. As of now, I thoroughly confused. >>> rich >>> >>> On Tuesday, 19 January 2021 at 15:55:38 UTC-5 garrya...@gmail.com wrote: >>> >>>> I wasn't happy leaving this is as is so I did some more testing. >>>> >>>> I looked in /home/weewx/bin/user/belchertown.py for any mention of the >>>> 4 include files it uses to add content - nothing there. >>>> >>>> So, I looked in /home/weewx/skins/Belchertown/index.html.templ and >>>> found 4 lines that reference the include files - they are of the form: >>>> >>>> #if os.path.exists("index_hook_after_station_info.inc") >>>> <!-- Start of index_hook_after_station_info row --> >>>> <div class="row index-hook-after-station-info >>>> border-bottom"> >>>> #include "index_hook_after_station_info.inc" >>>> </div> >>>> <!-- End of index_hook_after_station_info row --> >>>> #end if >>>> That caused me to look up the Cheetah '#include' directive: >>>> >>>> "7.6 #include >>>> >>>> Syntax: >>>> #include [raw] FILENAME_EXPR #include [raw] source=STRING_EXPR >>>> >>>> The #include directive is used to include text from outside the >>>> template definition. The text can come from an external file or from >>>> a $placeholder variable. When working with external files, Cheetah will >>>> monitor for changes to the included file and update as necessary. >>>> >>>> This example demonstrates its use with external files: >>>> #include "includeFileName.txt" >>>> The content of "includeFileName.txt" will be parsed for Cheetah syntax. >>>> >>>> And this example demonstrates use with $placeholder variables: >>>> #include source=$myParseText >>>> The value of $myParseText will be parsed for Cheetah syntax. This is >>>> not the same as simply placing the $placeholder tag ``$myParseText'' in >>>> the >>>> template definition. In the latter case, the value of $myParseText would >>>> not be parsed. >>>> >>>> By default, included text will be parsed for Cheetah tags. The argument >>>> ``raw'' can be used to suppress the parsing. >>>> >>>> #include raw "includeFileName.txt" #include raw source=$myParseText >>>> >>>> Cheetah wraps each chunk of #include text inside a >>>> nested Template object. Each nested template has a copy of the main >>>> template's searchList. However, #set variables are visible across includes >>>> only if the defined using the #set global keyword. >>>> >>>> All directives must be balanced in the include file. That is, if you >>>> start a #for or #if block inside the include, you must end it in the same >>>> include. (This is unlike PHP, which allows unbalanced constructs in >>>> include >>>> files.)" >>>> >>>> I decided to insert the 'raw' argument into the 4 '#include' directives >>>> to see if anything changed. >>>> >>>> After about 1 hour. memory usage did not grow above about 65 MB! >>>> >>>> I just removed the 'raw' argument and restarted WeeWX. After just a >>>> few minutes, memory usage is almost 300 MB and growing! >>>> >>>> As before, the station website is at: >>>> https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem graphs are >>>> at https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem. >>>> >>>> So, I think I have conclusively shown that the problem is within >>>> Cheetah and is related to it parsing an include file. >>>> >>>> I'm going to re-insert the 'raw' arguments and carry on. >>>> >>>> Should I report this issue to the Cheetah folks or is there someone >>>> closer to the Cheetah developers that can better do that? >>>> >>>> Regards, >>>> >>>> Garry >>>> >>>> >>>> On Monday, January 18, 2021 at 9:27:42 AM UTC-8 garrya...@gmail.com >>>> wrote: >>>> >>>>> Thanks! >>>>> >>>>> I think I've done all that I can to isolate the problem. >>>>> >>>>> I'm going to get back to finishing up development and add a cron entry >>>>> to restart WeeWX every 24 hours in the wee hours of the morning. >>>>> >>>>> I will be re-deploying a Davis Pro 2 system very soon (within the >>>>> week) and then deploying another Davis Pro 2 system shortly after that. >>>>> And sometime after than, a couple of WeatherFlow Tempest systems. >>>>> >>>>> I'll post everything on GitHub soonest. >>>>> >>>>> Regards, >>>>> >>>>> Garry >>>>> PS: Re: Ubuntu Desktop on RPi, I've been using it for development for >>>>> the last few days and it has not crashed or hung, so I think I'm going to >>>>> stay with it. Only issue is wireless mouse lag (Raspberry Pi OS had >>>>> same/similar problem) and I've ordered a Logitech MK540 wireless >>>>> keyboard/mouse combo to see if newer hardwaere works better (than my >>>>> dated >>>>> Microsoft mouse). >>>>> On Sunday, January 17, 2021 at 5:40:15 PM UTC-8 vince wrote: >>>>> >>>>>> Restarted with your add-on plus belchertown both enabled - the RSS is >>>>>> nominally growing at 4 MB per 5-minute archive period. >>>>>> >>>>>> 1610928917 139.53125 63.97265625 15.453125 >>>>>> 1610929217 143.02734375 70.41796875 15.4921875 >>>>>> 1610929517 144.46484375 74.43359375 15.4921875 >>>>>> 1610929817 145.96484375 78.48046875 15.55078125 >>>>>> 1610930117 147.46484375 82.65625 15.58203125 >>>>>> 1610930417 149.09375 86.69921875 15.58203125 >>>>>> 1610930717 150.34375 90.55078125 15.58203125 >>>>>> 1610931017 151.84375 94.203125 15.24609375 >>>>>> >>>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to weewx-user+...@googlegroups.com. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/4916786b-3bb2-49a6-93b1-87a60a822dbdn%40googlegroups.com.
