I haven’t made any changes yet but in thinking about your workaround, it will be interesting to see if Cheetah actuality processes the included include file or if it skips it because the outer include file’s attributes are static.
If Cheetah doesn’t process the inner include file, it still should be a way to inject the ‘raw’ argument without having to edit ‘index.html.tmpl’. That is, my code would generate both the outer and inner include files each cycle, with the outer include file containing the ‘raw’ argument as you suggest. Regards, Garry Lockyer C: +1.250.689.0686 E: ga...@lockyer.ca > On Jan 20, 2021, at 06:24, bell...@gmail.com <bellri...@gmail.com> wrote: > > Note, it is growing - just much slower. > >> On Wednesday, 20 January 2021 at 09:13:35 UTC-5 bell...@gmail.com wrote: >> Garry, >> I might have a workaround for you. Wrap your html file in a template. >> something like this. >> index.html.tmp (this is in the belchertown skin) >> #include index_hook_after_charts.inc >> >> index_hook_after_charts.inc (this is the 'wrapper', naturally if you 'owned' >> index.html.tmpl you would not need it) >> #include raw generated.html >> >> generated.html (the file you generate - name as you want) >> your html >> >> I ran this on my loop 10,000 times and saw no appreciable memory increase >> and performed better too. >> rich >> >>> On Tuesday, 19 January 2021 at 22:45:42 UTC-5 garrya...@gmail.com wrote: >>> Many thanks to all for poking at this! >>> >>> >>> Regards, >>> >>> Garry Lockyer >>> C: +1.250.689.0686 >>> E: ga...@lockyer.ca >>> >>> >>>>> On Jan 19, 2021, at 19:38, vince <vince...@gmail.com> wrote: >>>>> >>>> Just a followup - I took belchertown out of the picture and used my >>>> minimal demo-skin and added 'one line' that #include(d) a file generated >>>> by Garry's service, and it 'is' leaking memory albeit less quickly than >>>> the belchertown example..... >>> >>>> >>>> I'll let it run overnight and we'll see what it looks like after 12 more >>>> hours running. The VM is set to only 1GB RAM but I'm hoping if it runs >>>> out it'll just go into swap and stay up. OS is debian 10 under >>>> Vagrant/VirtualBox with 4.3.0 installed using python3 and setup.py >>>> >>>> Some mem.sdb output (date/time, units, interval, mem_size, mem_rss, >>>> mem_share)... >>>> >>>> 1611109216 16 5 141.0625 71.734375 12.2734375 >>>> 1611109516 16 5 142.5625 75.65234375 12.3359375 >>>> 1611109816 16 5 144.0625 79.65625 12.3359375 >>>> 1611110116 16 5 145.5625 84.5234375 12.3359375 >>>> 1611110416 16 5 147.24609375 88.0859375 12.3359375 >>>> 1611110716 16 5 148.49609375 91.43359375 12.3359375 >>>> 1611111016 16 5 149.99609375 95.421875 12.3359375 >>>> 1611111316 16 5 151.49609375 100.3359375 12.3359375 >>>> 1611111616 16 5 152.99609375 103.8359375 12.3359375 >>>> 1611111916 16 5 154.49609375 107.51171875 12.3359375 >>>> 1611112216 16 5 155.99609375 111.5 12.3359375 >>>> 1611112516 16 5 157.24609375 116.0859375 12.3359375 >>>> 1611112816 16 5 158.74609375 119.5859375 12.3359375 >>>> 1611113116 16 5 160.24609375 123.24609375 12.3359375 >>>> 1611113416 16 5 161.74609375 127.234375 12.3359375 >>>> 1611113716 16 5 163.24609375 132.11328125 12.3359375 >>>> >>>> >>>> >>>>> On Tuesday, January 19, 2021 at 5:21:31 PM UTC-8 bell...@gmail.com wrote: >>>>> Ok, my standalone testcase was wrong. I think I have a valid one here, >>>>> https://github.com/bellrichm/experiments >>>>> It just calls Cheetah.Template.Template in a loop. If the include file >>>>> modified attribute does not change, all seems stable. If it changes, >>>>> memory usage seems to grow. >>>>> rich >>>>> >>>>>> On Tuesday, 19 January 2021 at 19:16:46 UTC-5 tke...@gmail.com wrote: >>>>>> I'm not quite following this discussion. >>>>>> >>>>>> With Cheetah, you can compile the template and save the results. I >>>>>> believe this is what monitors the changed status of #include files. But, >>>>>> that's not the way WeeWX uses templates. It compiles, evaluates, then >>>>>> throws the results away. >>>>>> >>>>>> If you use the "raw" directive in an #include, then the file will not be >>>>>> parsed. So, how can any $ directives work? >>>>>> >>>>>> Or, does Belchertown work differently? >>>>>> >>>>>> -tk >>>>>> >>>>>>> On Tue, Jan 19, 2021 at 3:55 PM bell...@gmail.com <bell...@gmail.com> >>>>>>> wrote: >>>>>>> I agree that something does not seem quite right with Cheetah’s >>>>>>> ‘monitoring’ of include files. I took a different approach. >>>>>>> First, I wrote a simple WeeWX service that would perform the ‘touch’ >>>>>>> command against the large file and a simple template that included the >>>>>>> large file. When the file modified attribute changed, I observed the >>>>>>> memory growth, otherwise all was stable. Also, when the file was not >>>>>>> ‘touched’ the Cheetah processing was noticeably faster. >>>>>>> Next I wrote a standalone program to run a loop that ran the touch >>>>>>> command and invoke Cheetah. I ran it 3 times. Each time around loop >>>>>>> 90, the memory growth stopped and the processing was fast as though >>>>>>> Cheetah was using cached data. >>>>>>> I am back to running under WeeWX to see if it will level off like it >>>>>>> does standalone. I am also wading through the Cheetah code and >>>>>>> documentation. As of now, I thoroughly confused. >>>>>>> rich >>>>>>> >>>>>>>>> On Tuesday, 19 January 2021 at 15:55:38 UTC-5 garrya...@gmail.com >>>>>>>>> wrote: >>>>>>>>> I wasn't happy leaving this is as is so I did some more testing. >>>>>>>>> >>>>>>>>> I looked in /home/weewx/bin/user/belchertown.py for any mention of >>>>>>>>> the 4 include files it uses to add content - nothing there. >>>>>>>>> >>>>>>>>> So, I looked in /home/weewx/skins/Belchertown/index.html.templ and >>>>>>>>> found 4 lines that reference the include files - they are of the form: >>>>>>>>> >>>>>>>>> #if >>>>>>>>> os.path.exists("index_hook_after_station_info.inc") >>>>>>>>> <!-- Start of index_hook_after_station_info row --> >>>>>>>>> <div class="row index-hook-after-station-info >>>>>>>>> border-bottom"> >>>>>>>>> #include "index_hook_after_station_info.inc" >>>>>>>>> </div> >>>>>>>>> <!-- End of index_hook_after_station_info row --> >>>>>>>>> #end if >>>>>>>>> That caused me to look up the Cheetah '#include' directive: >>>>>>>>> >>>>>>>>> "7.6 #include >>>>>>>>> Syntax: >>>>>>>>> >>>>>>>>> #include [raw] FILENAME_EXPR #include [raw] source=STRING_EXPR >>>>>>>>> The #include directive is used to include text from outside the >>>>>>>>> template definition. The text can come from an external file or from >>>>>>>>> a $placeholder variable. When working with external files, Cheetah >>>>>>>>> will monitor for changes to the included file and update as necessary. >>>>>>>>> >>>>>>>>> This example demonstrates its use with external files: >>>>>>>>> >>>>>>>>> #include "includeFileName.txt" >>>>>>>>> The content of "includeFileName.txt" will be parsed for Cheetah >>>>>>>>> syntax. >>>>>>>>> And this example demonstrates use with $placeholder variables: >>>>>>>>> >>>>>>>>> #include source=$myParseText >>>>>>>>> The value of $myParseText will be parsed for Cheetah syntax. This is >>>>>>>>> not the same as simply placing the $placeholder tag ``$myParseText'' >>>>>>>>> in the template definition. In the latter case, the value of >>>>>>>>> $myParseText would not be parsed. >>>>>>>>> By default, included text will be parsed for Cheetah tags. The >>>>>>>>> argument ``raw'' can be used to suppress the parsing. >>>>>>>>> >>>>>>>>> #include raw "includeFileName.txt" #include raw source=$myParseText >>>>>>>>> Cheetah wraps each chunk of #include text inside a nested Template >>>>>>>>> object. Each nested template has a copy of the main template's >>>>>>>>> searchList. However, #set variables are visible across includes only >>>>>>>>> if the defined using the #set global keyword. >>>>>>>>> >>>>>>>>> All directives must be balanced in the include file. That is, if you >>>>>>>>> start a #for or #if block inside the include, you must end it in the >>>>>>>>> same include. (This is unlike PHP, which allows unbalanced constructs >>>>>>>>> in include files.)" >>>>>>>>> >>>>>>>>> I decided to insert the 'raw' argument into the 4 '#include' >>>>>>>>> directives to see if anything changed. >>>>>>>>> >>>>>>>>> After about 1 hour. memory usage did not grow above about 65 MB! >>>>>>>>> >>>>>>>>> I just removed the 'raw' argument and restarted WeeWX. After just a >>>>>>>>> few minutes, memory usage is almost 300 MB and growing! >>>>>>>>> >>>>>>>>> As before, the station website is at: >>>>>>>>> https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem graphs >>>>>>>>> are at https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem. >>>>>>>>> >>>>>>>>> So, I think I have conclusively shown that the problem is within >>>>>>>>> Cheetah and is related to it parsing an include file. >>>>>>>>> >>>>>>>>> I'm going to re-insert the 'raw' arguments and carry on. >>>>>>>>> >>>>>>>>> Should I report this issue to the Cheetah folks or is there someone >>>>>>>>> closer to the Cheetah developers that can better do that? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Garry >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Monday, January 18, 2021 at 9:27:42 AM UTC-8 garrya...@gmail.com >>>>>>>>>> wrote: >>>>>>>>>> Thanks! >>>>>>>>>> >>>>>>>>>> I think I've done all that I can to isolate the problem. >>>>>>>>>> >>>>>>>>>> I'm going to get back to finishing up development and add a cron >>>>>>>>>> entry to restart WeeWX every 24 hours in the wee hours of the >>>>>>>>>> morning. >>>>>>>>>> >>>>>>>>>> I will be re-deploying a Davis Pro 2 system very soon (within the >>>>>>>>>> week) and then deploying another Davis Pro 2 system shortly after >>>>>>>>>> that. And sometime after than, a couple of WeatherFlow Tempest >>>>>>>>>> systems. >>>>>>>>>> >>>>>>>>>> I'll post everything on GitHub soonest. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Garry >>>>>>>>>> PS: Re: Ubuntu Desktop on RPi, I've been using it for development >>>>>>>>>> for the last few days and it has not crashed or hung, so I think I'm >>>>>>>>>> going to stay with it. Only issue is wireless mouse lag (Raspberry >>>>>>>>>> Pi OS had same/similar problem) and I've ordered a Logitech MK540 >>>>>>>>>> wireless keyboard/mouse combo to see if newer hardwaere works better >>>>>>>>>> (than my dated Microsoft mouse). >>>>>>>>>>> On Sunday, January 17, 2021 at 5:40:15 PM UTC-8 vince wrote: >>>>>>>>>>> Restarted with your add-on plus belchertown both enabled - the RSS >>>>>>>>>>> is nominally growing at 4 MB per 5-minute archive period. >>>>>>>>>>> >>>>>>>>>>> 1610928917 139.53125 63.97265625 15.453125 >>>>>>>>>>> 1610929217 143.02734375 70.41796875 15.4921875 >>>>>>>>>>> 1610929517 144.46484375 74.43359375 15.4921875 >>>>>>>>>>> 1610929817 145.96484375 78.48046875 15.55078125 >>>>>>>>>>> 1610930117 147.46484375 82.65625 15.58203125 >>>>>>>>>>> 1610930417 149.09375 86.69921875 15.58203125 >>>>>>>>>>> 1610930717 150.34375 90.55078125 15.58203125 >>>>>>>>>>> 1610931017 151.84375 94.203125 15.24609375 >>>>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "weewx-user" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to weewx-user+...@googlegroups.com. >>>>>> >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to weewx-user+...@googlegroups.com. >>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/4916786b-3bb2-49a6-93b1-87a60a822dbdn%40googlegroups.com. >>>> <Screen Shot 2021-01-19 at 7.33.46 PM.png> > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/71e3dde7-0174-42d2-908e-1d6ea27b42ban%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/A4C62DCE-1CA3-45CF-BA83-AD4D018B1C72%40gmail.com.
