That will stop Cheetah from using its cache, but it will still be added to
sys.modules; so I think the memory will still grow over time.
I think (2) is very reasonable. If a skin author is dynamically creating a
template, they can just take the next step and create the final content.
That is
Tom Keffer schrieb am Montag, 26. Juni 2023 um 14:01:03 UTC+2:
That won't fix anything. Possible solutions:
No, that would not fix anything, but would warn users, that are not aware
of the problem.
import Cheetah.Template.Templace
Cheetah.Template.Template._CHEETAH_useCompilationCache =
That won't fix anything. Possible solutions:
1. Submit an issue to the Cheetah authors.
2. Adopt a best practice to not change template code.
3. Or, if you do, disable the Cheetah caching. Put this in the Belchertown
code somewhere:
import Cheetah.Template.Templace
Tom Keffer schrieb am Sonntag, 25. Juni 2023 um 23:22:50 UTC+2:
Nicely done, Rich. Fits the facts perfectly.
So we could include some monitoring of sys.modules in the core loop of
WeeWX and complain to the logs if there is such an issue?
--
You received this message because you are
Nicely done, Rich. Fits the facts perfectly.
On Sun, Jun 25, 2023 at 2:12 PM bell...@gmail.com
wrote:
> Your understanding is correct. As part of my debugging I ran the Seasons
> skin and some additional skins. After each template was processed I logged
> the number of entries in sys.modules.
Your understanding is correct. As part of my debugging I ran the Seasons
skin and some additional skins. After each template was processed I logged
the number of entries in sys.modules. On the first reporting cycle it
increased by 1 for each template. After that it has not changed.
Belchertown
Good sleuthing! Pretty crappy algorithm that allows a cache to grow without
bounds.
The skins that come with WeeWX (Seasons, Standard) use static source
template files. They are always the same. So, if I understand you
correctly, each template should take only one slot in sys.modules.
Does the
In my experimental skin I was running into this issue, so I decided to
delve into Cheetah’s internals and learned the following. As part of the
compilation process the module is added to sys.modules and the generated
code is cached. If one is dynamically generating template source and it is
The Cheetah folks have asked for a pure Cheetah method to reproduce the
problem, which I do not have.
Perhaps someone else can assist them.
The issue report is
at: https://github.com/CheetahTemplate3/cheetah3/issues/34
Regards,
Garry
On Wednesday, January 27, 2021 at 11:39:39 PM UTC-8
I just entered an issue against Cheetah3.
On Saturday, January 23, 2021 at 7:47:59 AM UTC-8 wes...@gmail.com wrote:
> has anyone filed an issue on the github project for cheetah?
>
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To
has anyone filed an issue on the github project for cheetah?
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this
Ok thank you Garry
On Fri, Jan 22, 2021, 7:33 PM Garry A Lockyer
wrote:
> I’ll assume your talking to me.
>
> There are no proposed changes to any existing software - at least not to
> WeeWX or the Belchertown skin. Any changes would likely be modifications
> to the Cheetah generator so that
I’ll assume your talking to me.
There are no proposed changes to any existing software - at least not to WeeWX
or the Belchertown skin. Any changes would likely be modifications to the
Cheetah generator so that it does not consume ever increasing amounts of
memory, but I know nothing
Would it be possible to post a step by step instruction for your changes if
not too much trouble?
On Fri, Jan 22, 2021, 3:01 PM Garry A Lockyer
wrote:
> The workaround is to have “static” index hook files that each just have 1
> Cheetah directive in them. That directive has the ‘raw’ argument.
The workaround is to have “static” index hook files that each just have 1
Cheetah directive in them. That directive has the ‘raw’ argument. For my
extension, I will implement the work around, using
“index-hook-after-station-info.inc” (and assuming I got the file name correct!)
as the
I'm tracking this thread a bit slowly... Is the raw portion within the
#include custom to your setup - or is this something that should be added
to the Belchertown skin?
On Thursday, January 21, 2021 at 12:47:26 PM UTC-5 garrya...@gmail.com
wrote:
> It looks like the workaround Rich suggested
It looks like the workaround Rich suggested works.
A couple of notes:
1. the generated filename must be in quotes (#include raw "generated file
name")
- the Cheetah generator will throw an error if not.
2. There must not be 'raw' arguments in index.html.tmpl
- I left them in after
Thanks to Vince, Rich and all who took the time to investigate this problem.
Happy that it’s not WeeWX or Belchertown, two fine programs!
I hope to release my WXFeeds service extension within the next few days.
Regards,
Garry
On Wednesday, January 20, 2021 at 8:19:32 AM UTC-8 vince wrote:
>
Well FWIW, it's not Belchertown.I took my minimalist demo-skin and aded
the one include line in it and boy does it grow.
Two tests with a restart that's obvious.
On Wednesday, January 20, 2021 at 7:07:23 AM UTC-8 garrya...@gmail.com
wrote:
> I haven’t made any changes yet but in
I haven’t made any changes yet but in thinking about your workaround, it will
be interesting to see if Cheetah actuality processes the included include file
or if it skips it because the outer include file’s attributes are static.
If Cheetah doesn’t process the inner include file, it still
Very clever! Thanks!!
I’ll give it a try in a few hours.
Regards,
Garry Lockyer
C: +1.250.689.0686
E: ga...@lockyer.ca
> On Jan 20, 2021, at 06:24, bell...@gmail.com wrote:
>
> Note, it is growing - just much slower.
>
>> On Wednesday, 20 January 2021 at 09:13:35 UTC-5 bell...@gmail.com
Note, it is growing - just much slower.
On Wednesday, 20 January 2021 at 09:13:35 UTC-5 bell...@gmail.com wrote:
> Garry,
> I might have a workaround for you. Wrap your html file in a template.
> something like this.
> index.html.tmp (this is in the belchertown skin)
> #include
Garry,
I might have a workaround for you. Wrap your html file in a template.
something like this.
index.html.tmp (this is in the belchertown skin)
#include index_hook_after_charts.inc
index_hook_after_charts.inc (this is the 'wrapper', naturally if you
'owned' index.html.tmpl you would not need
Many thanks to all for poking at this!
Regards,
Garry Lockyer
C: +1.250.689.0686
E: ga...@lockyer.ca
> On Jan 19, 2021, at 19:38, vince wrote:
>
> Just a followup - I took belchertown out of the picture and used my minimal
> demo-skin and added 'one line' that #include(d) a file generated
Just a followup - I took belchertown out of the picture and used my minimal
demo-skin and added 'one line' that #include(d) a file generated by Garry's
service, and it 'is' leaking memory albeit less quickly than the
belchertown example.
I'll let it run overnight and we'll see what it
Ok, my standalone testcase was wrong. I think I have a valid one
here, https://github.com/bellrichm/experiments
It just calls Cheetah.Template.Template in a loop. If the include file
modified attribute does not change, all seems stable. If it changes, memory
usage seems to grow.
rich
On
Indeed. Everybody else is equally surprised. But it's very reproducible
and repeatable.
On Tuesday, January 19, 2021 at 5:14:20 PM UTC-8 Tom Keffer wrote:
> OK, got it.
>
> Still, I'm surprised that memory would grow with a #include, raw, or not.
> As I say, compiled templates are thrown
OK, got it.
Still, I'm surprised that memory would grow with a #include, raw, or not.
As I say, compiled templates are thrown away after every use.
On Tue, Jan 19, 2021 at 5:08 PM Garry A Lockyer
wrote:
> With apologies if you know much of this. . .
>
> The Belchertown skin allows you to add
With apologies if you know much of this. . .
The Belchertown skin allows you to add content to the main page it emits each
archive cycle via 4 “hook” files. If the hook files are in Belchertown’s skin
folder, their content is included in ‘index.html’.
I first learned about this feature ‘cause
No, I think you're following fine.
In his unique case, there are no $ directives nor cheetah parsing of the
file being #include(d). He's pre-generating it in his service as a
complete html table and letting Belchertown #include that html in the right
place on the resulting index.html page.
I'm not quite following this discussion.
With Cheetah, you can compile the template and save the results. I believe
this is what monitors the changed status of #include files. But, that's not
the way WeeWX uses templates. It compiles, evaluates, then throws the
results away.
If you use the "raw"
I agree that something does not seem quite right with Cheetah’s
‘monitoring’ of include files. I took a different approach.
First, I wrote a simple WeeWX service that would perform the ‘touch’
command against the large file and a simple template that included the
large file. When the file
On Tuesday, January 19, 2021 at 1:49:43 PM UTC-8 garrya...@gmail.com wrote:
> I prefer restarting WeeWX daily over hand editing index.html.tmpl but
> that frequency may not work for all cases so I’ll probably live with
> editing.
>
>
You should be able to let weewx run literally years
Thanks for your reply.
I’m going to respectfully disagree though re: things working as intended. . .
Yes, the include files I’m playing with are huge but only for testing -
aggravating - the problem. The problem was first noticed with very much
smaller include files on a 512 MB system that
I'm thinking it's actually working as designed if you assume Pat intended
that cheetah directives should be parsed within a .inc(luded) file. You
just might be the edge case where you have a rather massive include that
does 'not' want cheetah directive parsing.
If you open an issue with
I wasn't happy leaving this is as is so I did some more testing.
I looked in /home/weewx/bin/user/belchertown.py for any mention of the 4
include files it uses to add content - nothing there.
So, I looked in /home/weewx/skins/Belchertown/index.html.templ and found 4
lines that reference the
Thanks!
I think I've done all that I can to isolate the problem.
I'm going to get back to finishing up development and add a cron entry to
restart WeeWX every 24 hours in the wee hours of the morning.
I will be re-deploying a Davis Pro 2 system very soon (within the week) and
then deploying
Restarted with your add-on plus belchertown both enabled - the RSS is
nominally growing at 4 MB per 5-minute archive period.
1610928917 139.53125 63.97265625 15.453125
1610929217 143.02734375 70.41796875 15.4921875
1610929517 144.46484375 74.43359375 15.4921875
1610929817 145.96484375
So you are seeing what I am - the leak is evident when belchertown is enabled
*and* the .inc file is regenerated, yes?
Regards,
Garry Lockyer
C: +1.250.689.0686
E: ga...@lockyer.ca
> On Jan 17, 2021, at 16:06, vince wrote:
>
> garry - I installed your thing and did not see a memory
garry - I installed your thing and did not see a memory increase 'without'
belchertown running, and but it 'is' leaking' it seems 'with' belchertown
1.2 installed. I'm running the simulator and ubuntu-2004 in a Vagrant VM
running python3 weewx 4.3.0 via setup.py - Here's some raw numbers with
Thanks for your assistance.
I used setup.py to install WeeWX.
Yes, the HTML file is huge but it really is just a test case. I suspect a
smaller feed that generates a smaller include file would cause a smaller leak
that wouldn’t be so obvious. My actual extension has filters to select items
On Sunday, January 17, 2021 at 12:04:21 PM UTC-8 garrya...@gmail.com wrote:
> RPi 4 - 8 GB
> Ubuntu Raspberry Pi OS 20.10
> Python3 - 3.8.6
> WeeWX 4.3
> Belchertown 1.2
> - Problem was first noticed on RPi Zero W with Raspberry Pi OS Lite.
> - Problem was reproduced
- If you disable Belchertown and 'enable' your custom code that generates
the include file, does memory grow ?
- Yes. I just started at test run with WXFeedsMemoryTest.py (attached
here) and Belchertown enabled.
The station website is at:
https://lockyer.ca/weather/OsoyoosLakeNorthEast
- If you disable Belchertown and 'enable' your custom code that
generates the include file, does memory grow ?
- Yes. I just started at test run with WXFeedsMemoryTest.py
(attached here) and Belchertown enabled. The station website is at:
On Saturday, January 9, 2021 at 8:02:51 PM UTC-8 garrya...@gmail.com wrote:
> My experience is that when I recreate an include file for Belchertown on
> each archive cycle, as an extension to Belchertown or as a service, memory
> usage increases with each cycle and eventually errors start.
>
I haven't managed to package the "latest & greatest" up for others but I
did do some testing.
It occurred to me that since my extension is now a service, not dependent
on a skin, I could turn off a skin to see if that changed the problem. So
I disabled the Belchertown skin and restarted
Thanks, will do but later today.
Regards,
Garry Lockyer
C: +1.250.689.0686
E: ga...@lockyer.ca
> On Jan 16, 2021, at 11:09, bell...@gmail.com wrote:
>
>
> Garry,
> If you want me to run something, just make it available and give me some
> instructions.
> rich
>> On Saturday, 16 January
Garry,
If you want me to run something, just make it available and give me some
instructions.
rich
On Saturday, 16 January 2021 at 12:25:33 UTC-5 garrya...@gmail.com wrote:
> I've added the mem extension. My test station, with a large number of
> feeds selected, is at:
I've added the mem extension. My test station, with a large number of
feeds selected, is at: https://lockyer.ca/weather/OsoyoosLakeNorthEast
mem graphs are at: https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem
Regards,
Garry
On Saturday, January 16, 2021 at 8:42:09 AM UTC-8
Brief Update: January 16, 2021:
I've pretty much finished moving my extension to a service.
I started WeeWX at about midnight and at about 8:20 AM, memory usage has
grown to over 941MB! Allocated blocks was at 4,245,722.
So growing memory usage problem exists.
Bill indicated above he ran my
My experience is that when I recreate an include file for Belchertown on each
archive cycle, as an extension to Belchertown or as a service, memory usage
increases with each cycle and eventually errors start. I’ve seen the error I
reported in this string and unresponsive systems. So, yes, I
If your memory usage is stable, is there a problem ?
It's pretty typical after a weewx startup for the usage to go up a bit then
level off nicely.
On Saturday, January 9, 2021 at 4:41:51 PM UTC-8 bell...@gmail.com wrote:
> Garry,
> I ran your test extension for a bit in my Ubuntu VM. Doesn't
Garry,
I ran your test extension for a bit in my Ubuntu VM. Doesn't appear to be
leaking.
16:42:24 {'mem_size': 338.53515625, 'mem_rss': 32.953125, 'mem_share':
11.28125}
16:47:38 {'mem_size': 423.80859375, 'mem_rss': 47.75, 'mem_share':
11.796875}
16:52:53 {'mem_size': 424.4140625,
Many, many thanks!
Before I wrote the service to investigate the problem, I wrote a standalone
program to eliminate all WeeWX components. It did NOT exhibit the problem.
Problem only seems to show up under WeeWX/Belchertown *and* if the include file
changes / is recreated.
Anyway, I really
Garry,
I took your WxFeedsMemory.py and made some modifications.
First, I changed the logging to simple print calls. this was just a
convenience for me to easily see what was going on.
Second, I stole some code from Vince’s mem extension,
Yup!
Regards,
Garry Lockyer
C: +1.250.689.0686
E: ga...@lockyer.ca
> On Jan 7, 2021, at 11:07, vince wrote:
>
> Are you closing the file every time after you write it ?
>
>> On Thursday, January 7, 2021 at 10:00:40 AM UTC-8 garrya...@gmail.com wrote:
>> After many hours of troubleshooting
Are you closing the file every time after you write it ?
On Thursday, January 7, 2021 at 10:00:40 AM UTC-8 garrya...@gmail.com wrote:
> After many hours of troubleshooting and testing, I think I have an idea
> what's happening.
>
> Background: I want to use weewx with the Belchertown skin and
After many hours of troubleshooting and testing, I think I have an idea
what's happening.
Background: I want to use weewx with the Belchertown skin and my extension
that reads numerous RSS/Atom feeds on a Pi Zero, with either a Davis Pro
V2 or Weatherflow weather station. I will eventually
Got MemoryError after about 9 hours after restart. Have removed cmon by
commenting out any mention of cmon in weewx.conf and restarted.
Regards,
Garry Lockyer
Former DEC Product Support Engineer :^)
Kepner-Tregoe Trained :^))
C: +1.250.689.0686
E: ga...@lockyer.ca
> On Dec 31, 2020, at
On Thursday, December 31, 2020 at 11:39:39 AM UTC-8 garrya...@gmail.com
wrote:
> Re: editing the Belchertown skin, nope haven’t touched it, *other than
> interfacing with it via the include files (as generated by my
> BelchertownWxFeeds extension)*. When all the endpoints (for testing) are
>
Thanks for the speedy responses!
A WeeWx restart seems to have fixed the problem for the moment.
Re: editing the Belchertown skin, nope haven’t touched it, other than
interfacing with it via the include files (as generated by my
BelchertownWxFeeds extension). When all the endpoints (for
I don't have anything to add to this conversation, except that it's curious
that after 10+ years of never seeing a Python MemoryError, this is the
second in two days.
Perhaps an update to an underlying library is leaking memory like crazy?
-tk
On Thu, Dec 31, 2020 at 10:55 AM vince wrote:
>
Cheetah is notoriously hard to debug, and Belchertown is a very complicated
skin under the hood. Are you sure you didn't edit the Belchertown skin ?
If not, I'm going to guess here that the driver emitted something that the
Belchertown skin didn't expect, or that it did 'not' emit something
I just restarted weewx and memory usage dropped to about 15%!
I failed to note it in original post but CPU utilization was probably at
about 50% consistently before weewx restart and dropped to <30% after
restart.
Regards,
Garry
On Thursday, December 31, 2020 at 8:10:09 AM UTC-8
64 matches
Mail list logo
