I'm tracking this thread a bit slowly... Is the raw portion within the #include custom to your setup - or is this something that should be added to the Belchertown skin?
On Thursday, January 21, 2021 at 12:47:26 PM UTC-5 garrya...@gmail.com wrote: > It looks like the workaround Rich suggested works. > > A couple of notes: > > 1. the generated filename must be in quotes (#include raw "generated file > name") > - the Cheetah generator will throw an error if not. > > 2. There must not be 'raw' arguments in index.html.tmpl > - I left them in after previous debugging > - if they are in, the Cheetah generator will not parse the generated > file and the Cheetah directive will > be display in the output HTML. > > I will be putting a control in my extension to enable/disable this > workaround. > > Again, thanks to Rich and others for helping with this. > > Regards, > > Garry > > On Wednesday, January 20, 2021 at 10:16:26 AM UTC-8 garrya...@gmail.com > wrote: > >> Thanks to Vince, Rich and all who took the time to investigate this >> problem. >> >> Happy that it’s not WeeWX or Belchertown, two fine programs! >> >> I hope to release my WXFeeds service extension within the next few days. >> >> Regards, >> >> Garry >> >> On Wednesday, January 20, 2021 at 8:19:32 AM UTC-8 vince wrote: >> >>> Well FWIW, it's not Belchertown. I took my minimalist demo-skin and >>> aded the one include line in it and boy does it grow..... >>> >>> Two tests with a restart that's obvious. >>> >>> On Wednesday, January 20, 2021 at 7:07:23 AM UTC-8 garrya...@gmail.com >>> wrote: >>> >>>> I haven’t made any changes yet but in thinking about your workaround, >>>> it will be interesting to see if Cheetah actuality processes the included >>>> include file or if it skips it because the outer include file’s attributes >>>> are static. >>>> >>>> If Cheetah doesn’t process the inner include file, it still should be a >>>> way to inject the ‘raw’ argument without having to edit ‘index.html.tmpl’. >>>> That is, my code would generate both the outer and inner include files >>>> each cycle, with the outer include file containing the ‘raw’ argument as >>>> you suggest. >>>> >>>> Regards, >>>> >>>> Garry Lockyer >>>> C: +1.250.689.0686 <(250)%20689-0686> >>>> E: ga...@lockyer.ca >>>> >>>> >>>> On Jan 20, 2021, at 06:24, bell...@gmail.com <bell...@gmail.com> wrote: >>>> >>>> Note, it is growing - just much slower. >>>> >>>> >>>> >>>> On Wednesday, 20 January 2021 at 09:13:35 UTC-5 bell...@gmail.com >>>> wrote: >>>> >>>>> Garry, >>>>> I might have a workaround for you. Wrap your html file in a template. >>>>> something like this. >>>>> index.html.tmp (this is in the belchertown skin) >>>>> #include index_hook_after_charts.inc >>>>> >>>>> index_hook_after_charts.inc (this is the 'wrapper', naturally if you >>>>> 'owned' index.html.tmpl you would not need it) >>>>> #include raw generated.html >>>>> >>>>> generated.html (the file you generate - name as you want) >>>>> your html >>>>> >>>>> I ran this on my loop 10,000 times and saw no appreciable memory >>>>> increase and performed better too. >>>>> rich >>>>> >>>>> On Tuesday, 19 January 2021 at 22:45:42 UTC-5 garrya...@gmail.com >>>>> wrote: >>>>> >>>>>> Many thanks to all for poking at this! >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> Garry Lockyer >>>>>> C: +1.250.689.0686 <(250)%20689-0686> >>>>>> E: ga...@lockyer.ca >>>>>> >>>>>> >>>>>> On Jan 19, 2021, at 19:38, vince <vince...@gmail.com> wrote: >>>>>> >>>>>> Just a followup - I took belchertown out of the picture and used my >>>>>> minimal demo-skin and added 'one line' that #include(d) a file generated >>>>>> by >>>>>> Garry's service, and it 'is' leaking memory albeit less quickly than the >>>>>> belchertown example..... >>>>>> >>>>>> >>>>>> I'll let it run overnight and we'll see what it looks like after 12 >>>>>> more hours running. The VM is set to only 1GB RAM but I'm hoping if it >>>>>> runs out it'll just go into swap and stay up. OS is debian 10 under >>>>>> Vagrant/VirtualBox with 4.3.0 installed using python3 and setup.py >>>>>> >>>>>> Some mem.sdb output (date/time, units, interval, mem_size, mem_rss, >>>>>> mem_share)... >>>>>> >>>>>> 1611109216 16 5 141.0625 71.734375 12.2734375 >>>>>> 1611109516 16 5 142.5625 75.65234375 12.3359375 >>>>>> 1611109816 16 5 144.0625 79.65625 12.3359375 >>>>>> 1611110116 16 5 145.5625 84.5234375 12.3359375 >>>>>> 1611110416 16 5 147.24609375 88.0859375 12.3359375 >>>>>> 1611110716 16 5 148.49609375 91.43359375 12.3359375 >>>>>> 1611111016 16 5 149.99609375 95.421875 12.3359375 >>>>>> 1611111316 16 5 151.49609375 100.3359375 12.3359375 >>>>>> 1611111616 16 5 152.99609375 103.8359375 12.3359375 >>>>>> 1611111916 16 5 154.49609375 107.51171875 12.3359375 >>>>>> 1611112216 16 5 155.99609375 111.5 12.3359375 >>>>>> 1611112516 16 5 157.24609375 116.0859375 12.3359375 >>>>>> 1611112816 16 5 158.74609375 119.5859375 12.3359375 >>>>>> 1611113116 16 5 160.24609375 123.24609375 12.3359375 >>>>>> 1611113416 16 5 161.74609375 127.234375 12.3359375 >>>>>> 1611113716 16 5 163.24609375 132.11328125 12.3359375 >>>>>> >>>>>> >>>>>> >>>>>> On Tuesday, January 19, 2021 at 5:21:31 PM UTC-8 bell...@gmail.com >>>>>> wrote: >>>>>> >>>>>>> Ok, my standalone testcase was wrong. I think I have a valid one >>>>>>> here, https://github.com/bellrichm/experiments >>>>>>> It just calls Cheetah.Template.Template in a loop. If the include >>>>>>> file modified attribute does not change, all seems stable. If it >>>>>>> changes, >>>>>>> memory usage seems to grow. >>>>>>> rich >>>>>>> >>>>>>> On Tuesday, 19 January 2021 at 19:16:46 UTC-5 tke...@gmail.com >>>>>>> wrote: >>>>>>> >>>>>>>> I'm not quite following this discussion. >>>>>>>> >>>>>>>> With Cheetah, you can compile the template and save the results. I >>>>>>>> believe this is what monitors the changed status of #include files. >>>>>>>> But, >>>>>>>> that's not the way WeeWX uses templates. It compiles, evaluates, then >>>>>>>> throws the results away. >>>>>>>> >>>>>>>> If you use the "raw" directive in an #include, then the file will >>>>>>>> not be parsed. So, how can any $ directives work? >>>>>>>> >>>>>>>> Or, does Belchertown work differently? >>>>>>>> >>>>>>>> -tk >>>>>>>> >>>>>>>> On Tue, Jan 19, 2021 at 3:55 PM bell...@gmail.com < >>>>>>>> bell...@gmail.com> wrote: >>>>>>>> >>>>>>>>> I agree that something does not seem quite right with Cheetah’s >>>>>>>>> ‘monitoring’ of include files. I took a different approach. >>>>>>>>> First, I wrote a simple WeeWX service that would perform the >>>>>>>>> ‘touch’ command against the large file and a simple template that >>>>>>>>> included >>>>>>>>> the large file. When the file modified attribute changed, I observed >>>>>>>>> the >>>>>>>>> memory growth, otherwise all was stable. Also, when the file was not >>>>>>>>> ‘touched’ the Cheetah processing was noticeably faster. >>>>>>>>> Next I wrote a standalone program to run a loop that ran the touch >>>>>>>>> command and invoke Cheetah. I ran it 3 times. Each time around loop >>>>>>>>> 90, >>>>>>>>> the memory growth stopped and the processing was fast as though >>>>>>>>> Cheetah was >>>>>>>>> using cached data. >>>>>>>>> I am back to running under WeeWX to see if it will level off like >>>>>>>>> it does standalone. I am also wading through the Cheetah code and >>>>>>>>> documentation. As of now, I thoroughly confused. >>>>>>>>> rich >>>>>>>>> >>>>>>>>> On Tuesday, 19 January 2021 at 15:55:38 UTC-5 garrya...@gmail.com >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I wasn't happy leaving this is as is so I did some more testing. >>>>>>>>>> >>>>>>>>>> I looked in /home/weewx/bin/user/belchertown.py for any mention >>>>>>>>>> of the 4 include files it uses to add content - nothing there. >>>>>>>>>> >>>>>>>>>> So, I looked in /home/weewx/skins/Belchertown/index.html.templ >>>>>>>>>> and found 4 lines that reference the include files - they are of the >>>>>>>>>> form: >>>>>>>>>> >>>>>>>>>> #if >>>>>>>>>> os.path.exists("index_hook_after_station_info.inc") >>>>>>>>>> <!-- Start of index_hook_after_station_info row >>>>>>>>>> --> >>>>>>>>>> <div class="row index-hook-after-station-info >>>>>>>>>> border-bottom"> >>>>>>>>>> #include "index_hook_after_station_info.inc" >>>>>>>>>> </div> >>>>>>>>>> <!-- End of index_hook_after_station_info row --> >>>>>>>>>> #end if >>>>>>>>>> That caused me to look up the Cheetah '#include' directive: >>>>>>>>>> >>>>>>>>>> "7.6 #include >>>>>>>>>> >>>>>>>>>> Syntax: >>>>>>>>>> #include [raw] FILENAME_EXPR #include [raw] source=STRING_EXPR >>>>>>>>>> >>>>>>>>>> The #include directive is used to include text from outside the >>>>>>>>>> template definition. The text can come from an external file or from >>>>>>>>>> a $placeholder variable. When working with external files, Cheetah >>>>>>>>>> will >>>>>>>>>> monitor for changes to the included file and update as necessary. >>>>>>>>>> >>>>>>>>>> This example demonstrates its use with external files: >>>>>>>>>> #include "includeFileName.txt" >>>>>>>>>> The content of "includeFileName.txt" will be parsed for Cheetah >>>>>>>>>> syntax. >>>>>>>>>> >>>>>>>>>> And this example demonstrates use with $placeholder variables: >>>>>>>>>> #include source=$myParseText >>>>>>>>>> The value of $myParseText will be parsed for Cheetah syntax. This >>>>>>>>>> is not the same as simply placing the $placeholder tag >>>>>>>>>> ``$myParseText'' in >>>>>>>>>> the template definition. In the latter case, the value of >>>>>>>>>> $myParseText >>>>>>>>>> would not be parsed. >>>>>>>>>> >>>>>>>>>> By default, included text will be parsed for Cheetah tags. The >>>>>>>>>> argument ``raw'' can be used to suppress the parsing. >>>>>>>>>> >>>>>>>>>> #include raw "includeFileName.txt" #include raw >>>>>>>>>> source=$myParseText >>>>>>>>>> >>>>>>>>>> Cheetah wraps each chunk of #include text inside a >>>>>>>>>> nested Template object. Each nested template has a copy of the main >>>>>>>>>> template's searchList. However, #set variables are visible across >>>>>>>>>> includes >>>>>>>>>> only if the defined using the #set global keyword. >>>>>>>>>> >>>>>>>>>> All directives must be balanced in the include file. That is, if >>>>>>>>>> you start a #for or #if block inside the include, you must end it in >>>>>>>>>> the >>>>>>>>>> same include. (This is unlike PHP, which allows unbalanced >>>>>>>>>> constructs in >>>>>>>>>> include files.)" >>>>>>>>>> >>>>>>>>>> I decided to insert the 'raw' argument into the 4 '#include' >>>>>>>>>> directives to see if anything changed. >>>>>>>>>> >>>>>>>>>> After about 1 hour. memory usage did not grow above about 65 MB! >>>>>>>>>> >>>>>>>>>> I just removed the 'raw' argument and restarted WeeWX. After >>>>>>>>>> just a few minutes, memory usage is almost 300 MB and growing! >>>>>>>>>> >>>>>>>>>> As before, the station website is at: >>>>>>>>>> https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem >>>>>>>>>> graphs are at https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem >>>>>>>>>> . >>>>>>>>>> >>>>>>>>>> So, I think I have conclusively shown that the problem is within >>>>>>>>>> Cheetah and is related to it parsing an include file. >>>>>>>>>> >>>>>>>>>> I'm going to re-insert the 'raw' arguments and carry on. >>>>>>>>>> >>>>>>>>>> Should I report this issue to the Cheetah folks or is there >>>>>>>>>> someone closer to the Cheetah developers that can better do that? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Garry >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Monday, January 18, 2021 at 9:27:42 AM UTC-8 >>>>>>>>>> garrya...@gmail.com wrote: >>>>>>>>>> >>>>>>>>>>> Thanks! >>>>>>>>>>> >>>>>>>>>>> I think I've done all that I can to isolate the problem. >>>>>>>>>>> >>>>>>>>>>> I'm going to get back to finishing up development and add a cron >>>>>>>>>>> entry to restart WeeWX every 24 hours in the wee hours of the >>>>>>>>>>> morning. >>>>>>>>>>> >>>>>>>>>>> I will be re-deploying a Davis Pro 2 system very soon (within >>>>>>>>>>> the week) and then deploying another Davis Pro 2 system shortly >>>>>>>>>>> after >>>>>>>>>>> that. And sometime after than, a couple of WeatherFlow Tempest >>>>>>>>>>> systems. >>>>>>>>>>> >>>>>>>>>>> I'll post everything on GitHub soonest. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Garry >>>>>>>>>>> PS: Re: Ubuntu Desktop on RPi, I've been using it for >>>>>>>>>>> development for the last few days and it has not crashed or hung, >>>>>>>>>>> so I >>>>>>>>>>> think I'm going to stay with it. Only issue is wireless mouse lag >>>>>>>>>>> (Raspberry Pi OS had same/similar problem) and I've ordered a >>>>>>>>>>> Logitech >>>>>>>>>>> MK540 wireless keyboard/mouse combo to see if newer hardwaere works >>>>>>>>>>> better >>>>>>>>>>> (than my dated Microsoft mouse). >>>>>>>>>>> On Sunday, January 17, 2021 at 5:40:15 PM UTC-8 vince wrote: >>>>>>>>>>> >>>>>>>>>>>> Restarted with your add-on plus belchertown both enabled - the >>>>>>>>>>>> RSS is nominally growing at 4 MB per 5-minute archive period. >>>>>>>>>>>> >>>>>>>>>>>> 1610928917 139.53125 63.97265625 15.453125 >>>>>>>>>>>> 1610929217 143.02734375 70.41796875 15.4921875 >>>>>>>>>>>> 1610929517 144.46484375 74.43359375 15.4921875 >>>>>>>>>>>> 1610929817 145.96484375 78.48046875 15.55078125 >>>>>>>>>>>> 1610930117 147.46484375 82.65625 15.58203125 >>>>>>>>>>>> 1610930417 149.09375 86.69921875 15.58203125 >>>>>>>>>>>> 1610930717 150.34375 90.55078125 15.58203125 >>>>>>>>>>>> 1610931017 151.84375 94.203125 15.24609375 >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "weewx-user" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to weewx-user+...@googlegroups.com. >>>>>>>>> >>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to weewx-user+...@googlegroups.com. >>>>>> >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-user/4916786b-3bb2-49a6-93b1-87a60a822dbdn%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/weewx-user/4916786b-3bb2-49a6-93b1-87a60a822dbdn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> <Screen Shot 2021-01-19 at 7.33.46 PM.png> >>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to weewx-user+...@googlegroups.com. >>>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/71e3dde7-0174-42d2-908e-1d6ea27b42ban%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/weewx-user/71e3dde7-0174-42d2-908e-1d6ea27b42ban%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/9d1ce1c1-3017-43f2-b66f-13c5637748e6n%40googlegroups.com.
