The workaround is to have “static” index hook files that each just have 1 Cheetah directive in them. That directive has the ‘raw’ argument. For my extension, I will implement the work around, using “index-hook-after-station-info.inc” (and assuming I got the file name correct!) as the example, such that:
index-hook-after-station-info.inc contains: #include raw “WXFeeds-index-hook-after-station-info.inc” and is not re-created. My extension will generate “WXFeeds-index-hook-after-station-info.inc” each archive cycle. I don’t think any changes to Belchertown are required other than perhaps a warning that memory usage can increase if the hooks are used without the workaround (until the Cheetah generator is fixed). Thanks for your time, attention and the Belchertown skin! Regards, Garry Lockyer C: +1.250.689.0686 E: ga...@lockyer.ca > On Jan 22, 2021, at 12:00, Pat <pobrien...@gmail.com> wrote: > > I'm tracking this thread a bit slowly... Is the raw portion within the > #include custom to your setup - or is this something that should be added to > the Belchertown skin? > >> On Thursday, January 21, 2021 at 12:47:26 PM UTC-5 garrya...@gmail.com wrote: >> It looks like the workaround Rich suggested works. >> >> A couple of notes: >> >> 1. the generated filename must be in quotes (#include raw "generated file >> name") >> - the Cheetah generator will throw an error if not. >> >> 2. There must not be 'raw' arguments in index.html.tmpl >> - I left them in after previous debugging >> - if they are in, the Cheetah generator will not parse the generated >> file and the Cheetah directive will >> be display in the output HTML. >> >> I will be putting a control in my extension to enable/disable this >> workaround. >> >> Again, thanks to Rich and others for helping with this. >> >> Regards, >> >> Garry >> >>> On Wednesday, January 20, 2021 at 10:16:26 AM UTC-8 garrya...@gmail.com >>> wrote: >>> Thanks to Vince, Rich and all who took the time to investigate this problem. >>> >>> Happy that it’s not WeeWX or Belchertown, two fine programs! >>> >>> I hope to release my WXFeeds service extension within the next few days. >>> >>> Regards, >>> >>> Garry >>> >>>> On Wednesday, January 20, 2021 at 8:19:32 AM UTC-8 vince wrote: >>>> Well FWIW, it's not Belchertown. I took my minimalist demo-skin and >>>> aded the one include line in it and boy does it grow..... >>>> >>>> Two tests with a restart that's obvious. >>>> >>>>> On Wednesday, January 20, 2021 at 7:07:23 AM UTC-8 garrya...@gmail.com >>>>> wrote: >>>>> I haven’t made any changes yet but in thinking about your workaround, it >>>>> will be interesting to see if Cheetah actuality processes the included >>>>> include file or if it skips it because the outer include file’s >>>>> attributes are static. >>>>> >>>>> If Cheetah doesn’t process the inner include file, it still should be a >>>>> way to inject the ‘raw’ argument without having to edit >>>>> ‘index.html.tmpl’. That is, my code would generate both the outer and >>>>> inner include files each cycle, with the outer include file containing >>>>> the ‘raw’ argument as you suggest. >>>>> >>>>> Regards, >>>>> >>>>> Garry Lockyer >>>>> C: +1.250.689.0686 >>>>> E: ga...@lockyer.ca >>>>> >>>>> >>>>>>> On Jan 20, 2021, at 06:24, bell...@gmail.com <bell...@gmail.com> wrote: >>>>>>> >>>>>> Note, it is growing - just much slower. >>>>> >>>>>> >>>>>> >>>>>>> On Wednesday, 20 January 2021 at 09:13:35 UTC-5 bell...@gmail.com wrote: >>>>>>> Garry, >>>>>>> I might have a workaround for you. Wrap your html file in a template. >>>>>>> something like this. >>>>>>> index.html.tmp (this is in the belchertown skin) >>>>>>> #include index_hook_after_charts.inc >>>>>>> >>>>>>> index_hook_after_charts.inc (this is the 'wrapper', naturally if you >>>>>>> 'owned' index.html.tmpl you would not need it) >>>>>>> #include raw generated.html >>>>>>> >>>>>>> generated.html (the file you generate - name as you want) >>>>>>> your html >>>>>>> >>>>>>> I ran this on my loop 10,000 times and saw no appreciable memory >>>>>>> increase and performed better too. >>>>>>> rich >>>>>>> >>>>>>>> On Tuesday, 19 January 2021 at 22:45:42 UTC-5 garrya...@gmail.com >>>>>>>> wrote: >>>>>>>> Many thanks to all for poking at this! >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Garry Lockyer >>>>>>>> C: +1.250.689.0686 >>>>>>>> E: ga...@lockyer.ca >>>>>>>> >>>>>>>> >>>>>>>>>> On Jan 19, 2021, at 19:38, vince <vince...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>> Just a followup - I took belchertown out of the picture and used my >>>>>>>>> minimal demo-skin and added 'one line' that #include(d) a file >>>>>>>>> generated by Garry's service, and it 'is' leaking memory albeit less >>>>>>>>> quickly than the belchertown example..... >>>>>>>> >>>>>>>>> >>>>>>>>> I'll let it run overnight and we'll see what it looks like after 12 >>>>>>>>> more hours running. The VM is set to only 1GB RAM but I'm hoping if >>>>>>>>> it runs out it'll just go into swap and stay up. OS is debian 10 >>>>>>>>> under Vagrant/VirtualBox with 4.3.0 installed using python3 and >>>>>>>>> setup.py >>>>>>>>> >>>>>>>>> Some mem.sdb output (date/time, units, interval, mem_size, mem_rss, >>>>>>>>> mem_share)... >>>>>>>>> >>>>>>>>> 1611109216 16 5 141.0625 71.734375 >>>>>>>>> 12.2734375 >>>>>>>>> 1611109516 16 5 142.5625 75.65234375 >>>>>>>>> 12.3359375 >>>>>>>>> 1611109816 16 5 144.0625 79.65625 >>>>>>>>> 12.3359375 >>>>>>>>> 1611110116 16 5 145.5625 84.5234375 >>>>>>>>> 12.3359375 >>>>>>>>> 1611110416 16 5 147.24609375 88.0859375 >>>>>>>>> 12.3359375 >>>>>>>>> 1611110716 16 5 148.49609375 91.43359375 >>>>>>>>> 12.3359375 >>>>>>>>> 1611111016 16 5 149.99609375 95.421875 >>>>>>>>> 12.3359375 >>>>>>>>> 1611111316 16 5 151.49609375 100.3359375 >>>>>>>>> 12.3359375 >>>>>>>>> 1611111616 16 5 152.99609375 103.8359375 >>>>>>>>> 12.3359375 >>>>>>>>> 1611111916 16 5 154.49609375 107.51171875 >>>>>>>>> 12.3359375 >>>>>>>>> 1611112216 16 5 155.99609375 111.5 12.3359375 >>>>>>>>> 1611112516 16 5 157.24609375 116.0859375 >>>>>>>>> 12.3359375 >>>>>>>>> 1611112816 16 5 158.74609375 119.5859375 >>>>>>>>> 12.3359375 >>>>>>>>> 1611113116 16 5 160.24609375 123.24609375 >>>>>>>>> 12.3359375 >>>>>>>>> 1611113416 16 5 161.74609375 127.234375 >>>>>>>>> 12.3359375 >>>>>>>>> 1611113716 16 5 163.24609375 132.11328125 >>>>>>>>> 12.3359375 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Tuesday, January 19, 2021 at 5:21:31 PM UTC-8 bell...@gmail.com >>>>>>>>>> wrote: >>>>>>>>>> Ok, my standalone testcase was wrong. I think I have a valid one >>>>>>>>>> here, https://github.com/bellrichm/experiments >>>>>>>>>> It just calls Cheetah.Template.Template in a loop. If the include >>>>>>>>>> file modified attribute does not change, all seems stable. If it >>>>>>>>>> changes, memory usage seems to grow. >>>>>>>>>> rich >>>>>>>>>> >>>>>>>>>>> On Tuesday, 19 January 2021 at 19:16:46 UTC-5 tke...@gmail.com >>>>>>>>>>> wrote: >>>>>>>>>>> I'm not quite following this discussion. >>>>>>>>>>> >>>>>>>>>>> With Cheetah, you can compile the template and save the results. I >>>>>>>>>>> believe this is what monitors the changed status of #include files. >>>>>>>>>>> But, that's not the way WeeWX uses templates. It compiles, >>>>>>>>>>> evaluates, then throws the results away. >>>>>>>>>>> >>>>>>>>>>> If you use the "raw" directive in an #include, then the file will >>>>>>>>>>> not be parsed. So, how can any $ directives work? >>>>>>>>>>> >>>>>>>>>>> Or, does Belchertown work differently? >>>>>>>>>>> >>>>>>>>>>> -tk >>>>>>>>>>> >>>>>>>>>>>> On Tue, Jan 19, 2021 at 3:55 PM bell...@gmail.com >>>>>>>>>>>> <bell...@gmail.com> wrote: >>>>>>>>>>>> I agree that something does not seem quite right with Cheetah’s >>>>>>>>>>>> ‘monitoring’ of include files. I took a different approach. >>>>>>>>>>>> First, I wrote a simple WeeWX service that would perform the >>>>>>>>>>>> ‘touch’ command against the large file and a simple template that >>>>>>>>>>>> included the large file. When the file modified attribute changed, >>>>>>>>>>>> I observed the memory growth, otherwise all was stable. Also, when >>>>>>>>>>>> the file was not ‘touched’ the Cheetah processing was noticeably >>>>>>>>>>>> faster. >>>>>>>>>>>> Next I wrote a standalone program to run a loop that ran the touch >>>>>>>>>>>> command and invoke Cheetah. I ran it 3 times. Each time around >>>>>>>>>>>> loop 90, the memory growth stopped and the processing was fast as >>>>>>>>>>>> though Cheetah was using cached data. >>>>>>>>>>>> I am back to running under WeeWX to see if it will level off like >>>>>>>>>>>> it does standalone. I am also wading through the Cheetah code and >>>>>>>>>>>> documentation. As of now, I thoroughly confused. >>>>>>>>>>>> rich >>>>>>>>>>>> >>>>>>>>>>>>>> On Tuesday, 19 January 2021 at 15:55:38 UTC-5 >>>>>>>>>>>>>> garrya...@gmail.com wrote: >>>>>>>>>>>>>> I wasn't happy leaving this is as is so I did some more testing. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I looked in /home/weewx/bin/user/belchertown.py for any mention >>>>>>>>>>>>>> of the 4 include files it uses to add content - nothing there. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So, I looked in /home/weewx/skins/Belchertown/index.html.templ >>>>>>>>>>>>>> and found 4 lines that reference the include files - they are of >>>>>>>>>>>>>> the form: >>>>>>>>>>>>>> >>>>>>>>>>>>>> #if >>>>>>>>>>>>>> os.path.exists("index_hook_after_station_info.inc") >>>>>>>>>>>>>> <!-- Start of index_hook_after_station_info row >>>>>>>>>>>>>> --> >>>>>>>>>>>>>> <div class="row index-hook-after-station-info >>>>>>>>>>>>>> border-bottom"> >>>>>>>>>>>>>> #include "index_hook_after_station_info.inc" >>>>>>>>>>>>>> </div> >>>>>>>>>>>>>> <!-- End of index_hook_after_station_info row --> >>>>>>>>>>>>>> #end if >>>>>>>>>>>>>> That caused me to look up the Cheetah '#include' directive: >>>>>>>>>>>>>> >>>>>>>>>>>>>> "7.6 #include >>>>>>>>>>>>>> Syntax: >>>>>>>>>>>>>> >>>>>>>>>>>>>> #include [raw] FILENAME_EXPR #include [raw] source=STRING_EXPR >>>>>>>>>>>>>> The #include directive is used to include text from outside the >>>>>>>>>>>>>> template definition. The text can come from an external file or >>>>>>>>>>>>>> from a $placeholder variable. When working with external files, >>>>>>>>>>>>>> Cheetah will monitor for changes to the included file and update >>>>>>>>>>>>>> as necessary. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This example demonstrates its use with external files: >>>>>>>>>>>>>> >>>>>>>>>>>>>> #include "includeFileName.txt" >>>>>>>>>>>>>> The content of "includeFileName.txt" will be parsed for Cheetah >>>>>>>>>>>>>> syntax. >>>>>>>>>>>>>> And this example demonstrates use with $placeholder variables: >>>>>>>>>>>>>> >>>>>>>>>>>>>> #include source=$myParseText >>>>>>>>>>>>>> The value of $myParseText will be parsed for Cheetah syntax. >>>>>>>>>>>>>> This is not the same as simply placing the $placeholder tag >>>>>>>>>>>>>> ``$myParseText'' in the template definition. In the latter case, >>>>>>>>>>>>>> the value of $myParseText would not be parsed. >>>>>>>>>>>>>> By default, included text will be parsed for Cheetah tags. The >>>>>>>>>>>>>> argument ``raw'' can be used to suppress the parsing. >>>>>>>>>>>>>> >>>>>>>>>>>>>> #include raw "includeFileName.txt" #include raw >>>>>>>>>>>>>> source=$myParseText >>>>>>>>>>>>>> Cheetah wraps each chunk of #include text inside a nested >>>>>>>>>>>>>> Template object. Each nested template has a copy of the main >>>>>>>>>>>>>> template's searchList. However, #set variables are visible >>>>>>>>>>>>>> across includes only if the defined using the #set global >>>>>>>>>>>>>> keyword. >>>>>>>>>>>>>> >>>>>>>>>>>>>> All directives must be balanced in the include file. That is, if >>>>>>>>>>>>>> you start a #for or #if block inside the include, you must end >>>>>>>>>>>>>> it in the same include. (This is unlike PHP, which allows >>>>>>>>>>>>>> unbalanced constructs in include files.)" >>>>>>>>>>>>>> >>>>>>>>>>>>>> I decided to insert the 'raw' argument into the 4 '#include' >>>>>>>>>>>>>> directives to see if anything changed. >>>>>>>>>>>>>> >>>>>>>>>>>>>> After about 1 hour. memory usage did not grow above about 65 MB! >>>>>>>>>>>>>> >>>>>>>>>>>>>> I just removed the 'raw' argument and restarted WeeWX. After >>>>>>>>>>>>>> just a few minutes, memory usage is almost 300 MB and growing! >>>>>>>>>>>>>> >>>>>>>>>>>>>> As before, the station website is at: >>>>>>>>>>>>>> https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem >>>>>>>>>>>>>> graphs are at >>>>>>>>>>>>>> https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So, I think I have conclusively shown that the problem is >>>>>>>>>>>>>> within Cheetah and is related to it parsing an include file. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm going to re-insert the 'raw' arguments and carry on. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Should I report this issue to the Cheetah folks or is there >>>>>>>>>>>>>> someone closer to the Cheetah developers that can better do that? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Garry >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Monday, January 18, 2021 at 9:27:42 AM UTC-8 >>>>>>>>>>>>>>> garrya...@gmail.com wrote: >>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think I've done all that I can to isolate the problem. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm going to get back to finishing up development and add a >>>>>>>>>>>>>>> cron entry to restart WeeWX every 24 hours in the wee hours of >>>>>>>>>>>>>>> the morning. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I will be re-deploying a Davis Pro 2 system very soon (within >>>>>>>>>>>>>>> the week) and then deploying another Davis Pro 2 system shortly >>>>>>>>>>>>>>> after that. And sometime after than, a couple of WeatherFlow >>>>>>>>>>>>>>> Tempest systems. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'll post everything on GitHub soonest. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Garry >>>>>>>>>>>>>>> PS: Re: Ubuntu Desktop on RPi, I've been using it for >>>>>>>>>>>>>>> development for the last few days and it has not crashed or >>>>>>>>>>>>>>> hung, so I think I'm going to stay with it. Only issue is >>>>>>>>>>>>>>> wireless mouse lag (Raspberry Pi OS had same/similar problem) >>>>>>>>>>>>>>> and I've ordered a Logitech MK540 wireless keyboard/mouse combo >>>>>>>>>>>>>>> to see if newer hardwaere works better (than my dated Microsoft >>>>>>>>>>>>>>> mouse). >>>>>>>>>>>>>>>> On Sunday, January 17, 2021 at 5:40:15 PM UTC-8 vince wrote: >>>>>>>>>>>>>>>> Restarted with your add-on plus belchertown both enabled - the >>>>>>>>>>>>>>>> RSS is nominally growing at 4 MB per 5-minute archive period. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1610928917 139.53125 63.97265625 15.453125 >>>>>>>>>>>>>>>> 1610929217 143.02734375 70.41796875 15.4921875 >>>>>>>>>>>>>>>> 1610929517 144.46484375 74.43359375 15.4921875 >>>>>>>>>>>>>>>> 1610929817 145.96484375 78.48046875 15.55078125 >>>>>>>>>>>>>>>> 1610930117 147.46484375 82.65625 15.58203125 >>>>>>>>>>>>>>>> 1610930417 149.09375 86.69921875 15.58203125 >>>>>>>>>>>>>>>> 1610930717 150.34375 90.55078125 15.58203125 >>>>>>>>>>>>>>>> 1610931017 151.84375 94.203125 15.24609375 >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>>>>> Groups "weewx-user" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>>>> send an email to weewx-user+...@googlegroups.com. >>>>>>>>>>> >>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "weewx-user" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to weewx-user+...@googlegroups.com. >>>>>>>> >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/weewx-user/4916786b-3bb2-49a6-93b1-87a60a822dbdn%40googlegroups.com. >>>>>>>>> <Screen Shot 2021-01-19 at 7.33.46 PM.png> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to weewx-user+...@googlegroups.com. >>>>> >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-user/71e3dde7-0174-42d2-908e-1d6ea27b42ban%40googlegroups.com. > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/9d1ce1c1-3017-43f2-b66f-13c5637748e6n%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/772028EA-5967-47A8-A130-5864076C10DD%40gmail.com.
