OK, got it.
Still, I'm surprised that memory would grow with a #include, raw, or not.
As I say, compiled templates are thrown away after every use.
On Tue, Jan 19, 2021 at 5:08 PM Garry A Lockyer <garryalock...@gmail.com>
wrote:
> With apologies if you know much of this. . .
>
> The Belchertown skin allows you to add content to the main page it emits
> each archive cycle via 4 “hook” files. If the hook files are in
> Belchertown’s skin folder, their content is included in ‘index.html’.
>
> I first learned about this feature ‘cause it is used by a
> WeeWX/Belchertown extension to fetch aviation weather observations (METARs)
> and forecasts (TAFS). That extension is a search list extension and
> eventually I used that mechanism to create my own extension to fetch METARs
> and TAFs, and extended it to fetch weather information (current conditions
> and forecasts) from Environment Canada and NOAA’s National Weather Service,
> Storm Prediction Center and National Hurricane Service, as well as road
> information from 511 Alberta, Rive BC and Washington State Department of
> Transportation.
>
> I deployed a Davis Pro 2 weather station with WeeWX, Belcherton and my
> extension and it crashed after about 3 days. After much debugging and
> rewriting (including converting my Belchertown specific search list
> extension to a skin independent service to create the include files, using
> Ubuntu instead of Raspberry Pi OS, and creating a much smaller/simpler
> program to demonstrate the problem), I narrowed the problem down to the
> Cheetah generator, specifically the parsing of included files by the
> Cheetah generator.
>
> With “out of the box” WeeWX and Belchertown, If the include files are
> recreated (each archive cycle) and the Belchertown skin is enabled, WeeWX
> memory usage will grow every cycle. If the ‘raw’ argument is added to the
> ‘include’ directives in Belcherotwn’s ‘index.html.tmpl’ memory usage
> stabilizes. The bigger the include files, the faster memory usage grows.
>
> My extension does not use $ directives in the include files it creates so
> losing that functionality is not a problem for me but obviously it could be
> an issue for other extensions or include file generators.
>
> I don’t think the Cheetah generator should cause increasing memory usage
> because it’s asked to include content, whether it’s asked to parse it or
> not. I can live with modifying ‘index.html.tmpl’ but hope the underlying
> problem gets fixed.
>
> I’ve entered as ‘issue’ against Belchertown so that its author is aware,
> can possibly provide a workaround in his code and/or “escalate” the issue
> to the Cheetah folks.
>
> Regards,
>
> Garry Lockyer
> C: +1.250.689.0686
> E: ga...@lockyer.ca
>
>
> On Jan 19, 2021, at 16:16, Tom Keffer <tkef...@gmail.com> wrote:
>
>
> I'm not quite following this discussion.
>
> With Cheetah, you can compile the template and save the results. I believe
> this is what monitors the changed status of #include files. But, that's not
> the way WeeWX uses templates. It compiles, evaluates, then throws the
> results away.
>
> If you use the "raw" directive in an #include, then the file will not be
> parsed. So, how can any $ directives work?
>
> Or, does Belchertown work differently?
>
> -tk
>
> On Tue, Jan 19, 2021 at 3:55 PM bell...@gmail.com <bellri...@gmail.com>
> wrote:
>
>> I agree that something does not seem quite right with Cheetah’s
>> ‘monitoring’ of include files. I took a different approach.
>> First, I wrote a simple WeeWX service that would perform the ‘touch’
>> command against the large file and a simple template that included the
>> large file. When the file modified attribute changed, I observed the memory
>> growth, otherwise all was stable. Also, when the file was not ‘touched’ the
>> Cheetah processing was noticeably faster.
>> Next I wrote a standalone program to run a loop that ran the touch
>> command and invoke Cheetah. I ran it 3 times. Each time around loop 90,
>> the memory growth stopped and the processing was fast as though Cheetah was
>> using cached data.
>> I am back to running under WeeWX to see if it will level off like it does
>> standalone. I am also wading through the Cheetah code and documentation.
>> As of now, I thoroughly confused.
>> rich
>>
>> On Tuesday, 19 January 2021 at 15:55:38 UTC-5 garrya...@gmail.com wrote:
>>
>>> I wasn't happy leaving this is as is so I did some more testing.
>>>
>>> I looked in /home/weewx/bin/user/belchertown.py for any mention of the 4
>>> include files it uses to add content - nothing there.
>>>
>>> So, I looked in /home/weewx/skins/Belchertown/index.html.templ and found
>>> 4 lines that reference the include files - they are of the form:
>>>
>>> #if os.path.exists("index_hook_after_station_info.inc")
>>> <!-- Start of index_hook_after_station_info row -->
>>> <div class="row index-hook-after-station-info
>>> border-bottom">
>>> #include "index_hook_after_station_info.inc"
>>> </div>
>>> <!-- End of index_hook_after_station_info row -->
>>> #end if
>>> That caused me to look up the Cheetah '#include' directive:
>>>
>>> "7.6 #include
>>>
>>> Syntax:
>>> #include [raw] FILENAME_EXPR #include [raw] source=STRING_EXPR
>>>
>>> The #include directive is used to include text from outside the template
>>> definition. The text can come from an external file or from
>>> a $placeholder variable. When working with external files, Cheetah will
>>> monitor for changes to the included file and update as necessary.
>>>
>>> This example demonstrates its use with external files:
>>> #include "includeFileName.txt"
>>> The content of "includeFileName.txt" will be parsed for Cheetah syntax.
>>>
>>> And this example demonstrates use with $placeholder variables:
>>> #include source=$myParseText
>>> The value of $myParseText will be parsed for Cheetah syntax. This is not
>>> the same as simply placing the $placeholder tag ``$myParseText'' in the
>>> template definition. In the latter case, the value of $myParseText would
>>> not be parsed.
>>>
>>> By default, included text will be parsed for Cheetah tags. The argument
>>> ``raw'' can be used to suppress the parsing.
>>>
>>> #include raw "includeFileName.txt" #include raw source=$myParseText
>>>
>>> Cheetah wraps each chunk of #include text inside a
>>> nested Template object. Each nested template has a copy of the main
>>> template's searchList. However, #set variables are visible across includes
>>> only if the defined using the #set global keyword.
>>>
>>> All directives must be balanced in the include file. That is, if you
>>> start a #for or #if block inside the include, you must end it in the same
>>> include. (This is unlike PHP, which allows unbalanced constructs in include
>>> files.)"
>>>
>>> I decided to insert the 'raw' argument into the 4 '#include' directives
>>> to see if anything changed.
>>>
>>> After about 1 hour. memory usage did not grow above about 65 MB!
>>>
>>> I just removed the 'raw' argument and restarted WeeWX. After just a few
>>> minutes, memory usage is almost 300 MB and growing!
>>>
>>> As before, the station website is at:
>>> https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem graphs are
>>> at https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem.
>>>
>>> So, I think I have conclusively shown that the problem is within
>>> Cheetah and is related to it parsing an include file.
>>>
>>> I'm going to re-insert the 'raw' arguments and carry on.
>>>
>>> Should I report this issue to the Cheetah folks or is there someone
>>> closer to the Cheetah developers that can better do that?
>>>
>>> Regards,
>>>
>>> Garry
>>>
>>>
>>> On Monday, January 18, 2021 at 9:27:42 AM UTC-8 garrya...@gmail.com
>>> wrote:
>>>
>>>> Thanks!
>>>>
>>>> I think I've done all that I can to isolate the problem.
>>>>
>>>> I'm going to get back to finishing up development and add a cron entry
>>>> to restart WeeWX every 24 hours in the wee hours of the morning.
>>>>
>>>> I will be re-deploying a Davis Pro 2 system very soon (within the week)
>>>> and then deploying another Davis Pro 2 system shortly after that. And
>>>> sometime after than, a couple of WeatherFlow Tempest systems.
>>>>
>>>> I'll post everything on GitHub soonest.
>>>>
>>>> Regards,
>>>>
>>>> Garry
>>>> PS: Re: Ubuntu Desktop on RPi, I've been using it for development for
>>>> the last few days and it has not crashed or hung, so I think I'm going to
>>>> stay with it. Only issue is wireless mouse lag (Raspberry Pi OS had
>>>> same/similar problem) and I've ordered a Logitech MK540 wireless
>>>> keyboard/mouse combo to see if newer hardwaere works better (than my dated
>>>> Microsoft mouse).
>>>> On Sunday, January 17, 2021 at 5:40:15 PM UTC-8 vince wrote:
>>>>
>>>>> Restarted with your add-on plus belchertown both enabled - the RSS is
>>>>> nominally growing at 4 MB per 5-minute archive period.
>>>>>
>>>>> 1610928917 139.53125 63.97265625 15.453125
>>>>> 1610929217 143.02734375 70.41796875 15.4921875
>>>>> 1610929517 144.46484375 74.43359375 15.4921875
>>>>> 1610929817 145.96484375 78.48046875 15.55078125
>>>>> 1610930117 147.46484375 82.65625 15.58203125
>>>>> 1610930417 149.09375 86.69921875 15.58203125
>>>>> 1610930717 150.34375 90.55078125 15.58203125
>>>>> 1610931017 151.84375 94.203125 15.24609375
>>>>>
>>>>> --
>> You received this message because you are subscribed to the Google Groups
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com
>> <https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/CAPq0zEBLSWhzeEFOyksrx64Pro2M0F2skgOCePZ0DUWSEoH7rw%40mail.gmail.com
> <https://groups.google.com/d/msgid/weewx-user/CAPq0zEBLSWhzeEFOyksrx64Pro2M0F2skgOCePZ0DUWSEoH7rw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/7F58A737-4F41-4C62-8C7F-A5CA7678C672%40gmail.com
> <https://groups.google.com/d/msgid/weewx-user/7F58A737-4F41-4C62-8C7F-A5CA7678C672%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/CAPq0zECVKS%3Dwo0bb8AxivhRkKLVkC6-b5UJpJLCc-uRy_oUcTw%40mail.gmail.com.
