Re: [weewx-user] Re: Second crash after 11 days
Well, that's no good. I've been doing some of my own tests here, and think that the weakrefs are an artifact of using pympler. I am running some experiments on my end with stretch. Unfortunately, I don't have a lot of time at the moment, so it may take a while. -tk On Sat, Feb 2, 2019 at 1:18 PM Steve2Q wrote: > Now up to 15% :( > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Now up to 15% :( -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Rats. It should have leveled off by now. On Fri, Feb 1, 2019 at 1:13 PM Steve2Q wrote: > Tom, so far it is climbing. Started at 3.4, this AM it was 7.7, and now 9.1 > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Tom, so far it is climbing. Started at 3.4, this AM it was 7.7, and now 9.1 -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
... and climbing? Or, has it been stable? On Fri, Feb 1, 2019 at 1:01 PM Steve2Q wrote: > Weewxd using 9.1% of memory as of now. > > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Weewxd using 9.1% of memory as of now. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
On Friday, February 1, 2019 at 7:29:27 AM UTC-8, Thomas Keffer wrote: > > Still, if you've got any early results... > > > yeah - this one is pretty interesting. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Still, if you've got any early results... On Fri, Feb 1, 2019 at 6:57 AM Steve2Q wrote: > Vince..good idea. I will let it keep running and chart the results as I > did last time. Hopefully it will level off. > >> >> Steve - give it 2-3 days since you've previously seen it grow quickly. >> What the heck...have a weekend maybe and see what it looks like Sunday or >> Monday. It should stay up that long regardless based on your past history, >> eh ? >> >> >> -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Vince..good idea. I will let it keep running and chart the results as I did last time. Hopefully it will level off. > > Steve - give it 2-3 days since you've previously seen it grow quickly. > What the heck...have a weekend maybe and see what it looks like Sunday or > Monday. It should stay up that long regardless based on your past history, > eh ? > > > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
On Thursday, January 31, 2019 at 4:48:43 PM UTC-8, Thomas Keffer wrote: > > It should go up, but stabilize within a couple hours. If you let it run > overnight, you should have a pretty good indication by morning. > > -tk > > On Thu, Jan 31, 2019 at 4:36 PM Steve2Q > > wrote: > >> Tom: will the memory usage stay fairly flat if things are working >> properly? I seem to remember some comments that it can go up, but then >> levels off. Is there some average % of usage that is considered "normal"? >> >> Steve >> >> >> Steve - give it 2-3 days since you've previously seen it grow quickly. What the heck...have a weekend maybe and see what it looks like Sunday or Monday. It should stay up that long regardless based on your past history, eh ? -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
It should go up, but stabilize within a couple hours. If you let it run overnight, you should have a pretty good indication by morning. -tk On Thu, Jan 31, 2019 at 4:36 PM Steve2Q wrote: > Tom: will the memory usage stay fairly flat if things are working > properly? I seem to remember some comments that it can go up, but then > levels off. Is there some average % of usage that is considered "normal"? > > Steve > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Tom: will the memory usage stay fairly flat if things are working properly? I seem to remember some comments that it can go up, but then levels off. Is there some average % of usage that is considered "normal"? Steve -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Several things: - The climb in the number of weakrefs, which are not used by WeeWX, so must be used indirectly in a library or driver; - Of the libraries used by WeeWX, only the usb drivers use weakrefs (I checked); - User Kurt has an Ultimeter, but with a serial connection (instead of usb), and has no such problems; - Serial cables are notorious for hardware and software problems; - There isn't much left! But, I could well be wrong! Hopefully, this stab-in-the-dark will work out. -tk On Thu, Jan 31, 2019 at 1:40 PM Steve2Q wrote: > I substituted another cable. This is the result of lsusb for this one: > > > With the reboot, top shows 3.4%, will watch it over the next 24 hours. > Just for info; what makes you suspect the cable? > > Steve > > > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
I substituted another cable. This is the result of lsusb for this one: With the reboot, top shows 3.4%, will watch it over the next 24 hours. Just for info; what makes you suspect the cable? Steve -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
You can comment out the debug_memory option. I'm suspecting your serial-to-usb cable. Do you have another one you can try? -tk On Thu, Jan 31, 2019 at 10:45 AM Steve2Q wrote: > Tom and Vince: > > Should I leave Weewx running as is or should I comment out the > debug_memory = True line, and switch back to the original engine.py? > > I have cut and pasted the information you asked for, Tom. If it ends up > being the driver, I could always use an old version I have (11rc3) and see > if that is any better. I will say that I was never sure why the driver > changed. The old one seemed to work fine, and the new one does not correct > the console clock like the old (I had posted about this in the past). > > Result of lsusb: > > > Result of dmesg: > > > > > > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Tom and Vince: Should I leave Weewx running as is or should I comment out the debug_memory = True line, and switch back to the original engine.py? I have cut and pasted the information you asked for, Tom. If it ends up being the driver, I could always use an old version I have (11rc3) and see if that is any better. I will say that I was never sure why the driver changed. The old one seemed to work fine, and the new one does not correct the console clock like the old (I had posted about this in the past). Result of lsusb: Result of dmesg: -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Good idea. I've sent a note to a couple of them. -tk On Thu, Jan 31, 2019 at 9:27 AM vince wrote: > On Thursday, January 31, 2019 at 8:00:11 AM UTC-8, Thomas Keffer wrote: >> >> It appears that "weakref" references are steadily climbing over time. >> Weak references are used to aid garbage collection in Python. They are not >> used in WeeWX, so they are probably being used by an underlying library. My >> candidate is the driver for your Ultimeter. >> >> >> > The map shows 14 Ultimeter stations registered currently. > Might be interesting to see if any of those stations are showing similar > behavior... > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
On Thursday, January 31, 2019 at 8:00:11 AM UTC-8, Thomas Keffer wrote: > > It appears that "weakref" references are steadily climbing over time. > Weak references are used to aid garbage collection in Python. They are not > used in WeeWX, so they are probably being used by an underlying library. My > candidate is the driver for your Ultimeter. > > > The map shows 14 Ultimeter stations registered currently. Might be interesting to see if any of those stations are showing similar behavior... -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
It appears that "weakref" references are steadily climbing over time. Weak references are used to aid garbage collection in Python. They are not used in WeeWX, so they are probably being used by an underlying library. My candidate is the driver for your Ultimeter. Could you please do two things? 1, First, run the command lsusb, then cut and paste the results. 2. Then unplug, then plug back in your Ultimeter, then run the command 'dmesg'. Cut and paste the last 20 lines or so that it prints out. It will look something like this: [93723.169773] usb 2-2: new full-speed USB device number 29 using xhci_hcd [93723.318669] usb 2-2: New USB device found, idVendor=05ad, idProduct=0fba [93723.318676] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [93723.318680] usb 2-2: Product: USB-Serial Controller [93723.318684] usb 2-2: Manufacturer: Prolific Technology Inc. [93723.319500] pl2303 2-2:1.0: pl2303 converter detected [93723.322547] usb 2-2: pl2303 converter now attached to ttyUSB0 This will tell us what modules are used by your Ultimeter. -tk On Thu, Jan 31, 2019 at 6:28 AM Steve2Q wrote: > Tom: attached is the summary file. Weewx has been running for 24 hours. > Weewxd is using 9.6% > > Steve > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Tom and Glenn..thanks..it is running now. I will let it run for a full 24 hours and post the results tomorrow. At this moment (weewx running for 5 minutes), top shows 4.6% memory usage by weewxd. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Glenn..it did say that it installed, but i will try your suggestion tomorrow. Thanks -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Hi Steve, The error is "No module named pympler" Looks like you've either missed step 1. of Toms instructions, or it's failed to install. Try installing it with sudo sudo pip install pympler On 30/01/2019, Steve2Q wrote: > Tom; the previous is what I got just by stopping weewx and then restarting > after the mods (no reboot of the pi). > >> If I reboot the pi, it hangs here: > > > I reverted back to the original setting so the station will keep running > until I hear back from you. > > Steve > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Cheers Glenn rorpi - read only raspberry pi & various weewx addons https://github.com/glennmckechnie -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Tom; the previous is what I got just by stopping weewx and then restarting after the mods (no reboot of the pi). > If I reboot the pi, it hangs here: I reverted back to the original setting so the station will keep running until I hear back from you. Steve -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Tom: I followed the instruction I received the following: -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
OK, Steve, I think we're ready. This is going to take a little preparation
on your part.
1. Install the tool pympler. This is a memory profiler.
*pip install pympler*
2. Replace your version of engine.py with the attached version. You should
find it in /home/weewx/bin/weewx/engine.py.
3. Edit your version of weewx.conf and add the highlighted line near the
top, under the 'debug' option:
# This section is for general configuration information.
# Set to 1 for extra debug info, otherwise comment it out or set to zero
debug = 0
debug_memory = True
# Root directory of the weewx data file hierarchy for this station
WEEWX_ROOT = /home/weewx
4. Run weewxd normally. It will profile memory after every archive
interval, and add the results to /var/tmp/weewx_memory_summary.
5. Let it run overnight or, at least, long enough that you can see memory
climbing. Post the file /var/tmp/weewx_memory_summary.
Let me know if you have any questions or problems.
-tk
On Tue, Jan 29, 2019 at 11:08 AM rich T wrote:
> Steve
>
> I'm running with desktop.
>
> On Tuesday, January 29, 2019 at 8:05:08 AM UTC-5, Steve2Q wrote:
>
>> Andrew and Rich...are you running the version of Stretch with the
>> desktop, of just the Lite version?
>>
>> My system did crash last evening as I thought it would.
>>
>> Steve
>>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
#
#Copyright (c) 2009-2015 Tom Keffer
#
#See the file LICENSE.txt for your full rights.
#
"""Main engine for the weewx weather system."""
# Python imports
import gc
import locale
import os
import os.path
import platform
import signal
import socket
import sys
import syslog
import time
import thread
# 3rd party imports:
import configobj
import daemon
# weewx imports:
import weedb
import weewx.accum
import weewx.manager
import weewx.qc
import weewx.station
import weewx.reportengine
import weeutil.weeutil
from weeutil.weeutil import to_bool, to_int, to_sorted_string
from weewx import all_service_groups
class BreakLoop(Exception):
"""Exception raised when it's time to break the main loop."""
class InitializationError(weewx.WeeWxIOError):
"""Exception raised when unable to initialize the console."""
#==
#Class StdEngine
#==
class StdEngine(object):
"""The main engine responsible for the creating and dispatching of events
from the weather station.
It loads a set of services, specified by an option in the configuration
file.
When a service loads, it binds callbacks to events. When an event occurs,
the bound callback will be called."""
def __init__(self, config_dict):
"""Initialize an instance of StdEngine.
config_dict: The configuration dictionary. """
self.debug_memory = to_bool(config_dict.get('debug_memory', False))
# Set a default socket time out, in case FTP or HTTP hang:
timeout = int(config_dict.get('socket_timeout', 20))
socket.setdefaulttimeout(timeout)
# Default garbage collection is every 3 hours:
self.gc_interval = int(config_dict.get('gc_interval', 3 * 3600))
# Set up the callback dictionary:
self.callbacks = dict()
# Set up the weather station hardware:
self.setupStation(config_dict)
# Hook for performing any chores before loading the services:
self.preLoadServices(config_dict)
# Load the services:
self.loadServices(config_dict)
# Another hook for after the services load.
self.postLoadServices(config_dict)
def setupStation(self, config_dict):
"""Set up the weather station hardware."""
# Get the hardware type from the configuration dictionary. This will be
# a string such as "VantagePro"
stationType = config_dict['Station']['station_type']
# Find the driver name for this type of hardware
driver = config_dict[stationType]['driver']
syslog.syslog(syslog.LOG_INFO, "engine: Loading station type %s (%s)" %
(stationType, driver))
# Import the driver:
__import__(driver)
# Open up the weather station, wrapping it in a try block in case
# of failure.
try:
# This is a bit of Python wizardry.
[weewx-user] Re: Second crash after 11 days
Steve I'm running with desktop. On Tuesday, January 29, 2019 at 8:05:08 AM UTC-5, Steve2Q wrote: > Andrew and Rich...are you running the version of Stretch with the desktop, > of just the Lite version? > > My system did crash last evening as I thought it would. > > Steve > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
I am not concerned about any crontab extensions. It's weewx.conf extensions that we care about. Can you please run *cd /home/weewx* *./bin/wee_debug --info --output* then email me (tkef...@gmail.com) the file /var/tmp/weewx.debug. On Tue, Jan 29, 2019 at 6:40 AM Steve2Q wrote: > Tom: I am running Stretch on a RPi 3 + (most recent Raspian from > raspberrypi.org; the version that includes desktop however I have the pi > set to boot into the cli, not the GUI). I have run the necessary commands > to update, upgrade, and update the firmware. I used setup.py with the > Ultimeter station, and except for a cron to backup .sdb, there are no other > extensions installed. If you need to look at any particular files, please > let me know. > >> >> -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
Tom: I am running Stretch on a RPi 3 + (most recent Raspian from raspberrypi.org; the version that includes desktop however I have the pi set to boot into the cli, not the GUI). I have run the necessary commands to update, upgrade, and update the firmware. I used setup.py with the Ultimeter station, and except for a cron to backup .sdb, there are no other extensions installed. If you need to look at any particular files, please let me know. > > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [weewx-user] Re: Second crash after 11 days
If I understand the situation correctly, you - Are running plain-vanilla wheezy on a stock RPi; - Installed weewx using setup.py; - Are using an Ultimeter station; - Have absolutely no extensions installed (in particular, rtg has been disabled); yet are experiencing memory growth. Please correct me if any of these assumptions are wrong. If they are correct, I'll make up an instrumented version of weewxd that will show which objects are growing in size and number. -tk On Tue, Jan 29, 2019 at 5:20 AM Steve2Q wrote: > Here is syslog from shortly before the crash.. Is there any other files > that may be useful for analysis? > > Jan 28 23:44:15 raspberrypi weewx[9770]: manager: Added record 2019-01-28 > 23:44:00 EST (1548737040) to database 'weewx.sdb' > Jan 28 23:44:16 raspberrypi weewx[9770]: manager: Added record 2019-01-28 > 23:44:00 EST (1548737040) to daily summary in 'weewx.sdb' > Jan 28 23:44:18 raspberrypi weewx[9770]: restx: PWSWeather: Published > record 2019-01-28 23:44:00 EST (1548737040) > Jan 28 23:44:26 raspberrypi weewx[9770]: cheetahgenerator: Generated 14 > files for report StandardReport in 8.67 seconds > Jan 28 23:44:36 raspberrypi weewx[9770]: imagegenerator: Generated 13 > images for StandardReport in 9.49 seconds > Jan 28 23:44:36 raspberrypi weewx[9770]: copygenerator: copied 0 files to > /home/weewx/public_html > Jan 28 23:44:47 raspberrypi weewx[9770]: ftpgenerator: ftp'd 27 files in > 11.38 seconds > Jan 28 23:46:15 raspberrypi weewx[9770]: manager: Added record 2019-01-28 > 23:46:00 EST (1548737160) to database 'weewx.sdb' > Jan 28 23:46:16 raspberrypi weewx[9770]: manager: Added record 2019-01-28 > 23:46:00 EST (1548737160) to daily summary in 'weewx.sdb' > Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.service: Failed to fork: > Cannot allocate memory > Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.service: Failed to run > 'start' task: Cannot allocate memory > Jan 28 23:46:39 raspberrypi systemd[1]: Failed to start Daily apt download > activities. > Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.timer: Adding 1h 13min > 18.732028s random time. > Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.service: Unit entered > failed state. > Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.timer: Adding 9h 23min > 52.201118s random time. > Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.service: Failed with > result 'resources'. > Jan 28 23:48:31 raspberrypi weewx[9770]: engine: Garbage collected 248392 > objects > Jan 28 23:48:32 raspberrypi kernel: [3815265.633466] top invoked > oom-killer: gfp_mask=0x14040d0(GFP_KERNEL|__GFP_COMP|__GFP_RECLAIMABLE), > nodemask=(null), order=0, oom_score_adj=0 > Jan 28 23:48:32 raspberrypi kernel: [3815265.633480] top cpuset=/ > mems_allowed=0 > Jan 28 23:48:32 raspberrypi kernel: [3815265.633495] CPU: 3 PID: 17293 > Comm: top Tainted: G C 4.14.87-v7+ #1178 > Jan 28 23:48:32 raspberrypi kernel: [3815265.633497] Hardware name: BCM2835 > Jan 28 23:48:32 raspberrypi kernel: [3815265.633521] [<8010ff30>] > (unwind_backtrace) from [<8010c174>] (show_stack+0x20/0x24) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633531] [<8010c174>] > (show_stack) from [<8078b424>] (dump_stack+0xd4/0x118) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633543] [<8078b424>] > (dump_stack) from [<80224bac>] (dump_header+0xac/0x208) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633553] [<80224bac>] > (dump_header) from [<80223f14>] (oom_kill_process+0x478/0x584) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633560] [<80223f14>] > (oom_kill_process) from [<80224874>] (out_of_memory+0x124/0x334) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633570] [<80224874>] > (out_of_memory) from [<8022a3b8>] (__alloc_pages_nodemask+0x10cc/0x11c0) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633579] [<8022a3b8>] > (__alloc_pages_nodemask) from [<80275a60>] (new_slab+0x454/0x558) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633587] [<80275a60>] > (new_slab) from [<802778a4>] (___slab_alloc.constprop.11+0x228/0x2c0) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633595] [<802778a4>] > (___slab_alloc.constprop.11) from [<80277980>] > (__slab_alloc.constprop.10+0x44/0x90) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633602] [<80277980>] > (__slab_alloc.constprop.10) from [<80278118>] (kmem_cache_alloc+0x1f4/0x230) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633611] [<80278118>] > (kmem_cache_alloc) from [<802f764c>] (proc_alloc_inode+0x2c/0x5c) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633622] [<802f764c>] > (proc_alloc_inode) from [<802a7ca8>] (alloc_inode+0x2c/0xb4) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633631] [<802a7ca8>] > (alloc_inode) from [<802aa01c>] (new_inode_pseudo+0x18/0x5c) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633639] [<802aa01c>] > (new_inode_pseudo) from [<802aa07c>] (new_inode+0x1c/0x30) > Jan 28 23:48:32 raspberrypi kernel: [3815265.633646] [<802aa07c>] > (new_inode) from [<802fb850>] (proc_pi
[weewx-user] Re: Second crash after 11 days
Here is syslog from shortly before the crash.. Is there any other files that may be useful for analysis? Jan 28 23:44:15 raspberrypi weewx[9770]: manager: Added record 2019-01-28 23:44:00 EST (1548737040) to database 'weewx.sdb' Jan 28 23:44:16 raspberrypi weewx[9770]: manager: Added record 2019-01-28 23:44:00 EST (1548737040) to daily summary in 'weewx.sdb' Jan 28 23:44:18 raspberrypi weewx[9770]: restx: PWSWeather: Published record 2019-01-28 23:44:00 EST (1548737040) Jan 28 23:44:26 raspberrypi weewx[9770]: cheetahgenerator: Generated 14 files for report StandardReport in 8.67 seconds Jan 28 23:44:36 raspberrypi weewx[9770]: imagegenerator: Generated 13 images for StandardReport in 9.49 seconds Jan 28 23:44:36 raspberrypi weewx[9770]: copygenerator: copied 0 files to /home/weewx/public_html Jan 28 23:44:47 raspberrypi weewx[9770]: ftpgenerator: ftp'd 27 files in 11.38 seconds Jan 28 23:46:15 raspberrypi weewx[9770]: manager: Added record 2019-01-28 23:46:00 EST (1548737160) to database 'weewx.sdb' Jan 28 23:46:16 raspberrypi weewx[9770]: manager: Added record 2019-01-28 23:46:00 EST (1548737160) to daily summary in 'weewx.sdb' Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.service: Failed to fork: Cannot allocate memory Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.service: Failed to run 'start' task: Cannot allocate memory Jan 28 23:46:39 raspberrypi systemd[1]: Failed to start Daily apt download activities. Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.timer: Adding 1h 13min 18.732028s random time. Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.service: Unit entered failed state. Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.timer: Adding 9h 23min 52.201118s random time. Jan 28 23:46:39 raspberrypi systemd[1]: apt-daily.service: Failed with result 'resources'. Jan 28 23:48:31 raspberrypi weewx[9770]: engine: Garbage collected 248392 objects Jan 28 23:48:32 raspberrypi kernel: [3815265.633466] top invoked oom-killer: gfp_mask=0x14040d0(GFP_KERNEL|__GFP_COMP|__GFP_RECLAIMABLE), nodemask=(null), order=0, oom_score_adj=0 Jan 28 23:48:32 raspberrypi kernel: [3815265.633480] top cpuset=/ mems_allowed=0 Jan 28 23:48:32 raspberrypi kernel: [3815265.633495] CPU: 3 PID: 17293 Comm: top Tainted: G C 4.14.87-v7+ #1178 Jan 28 23:48:32 raspberrypi kernel: [3815265.633497] Hardware name: BCM2835 Jan 28 23:48:32 raspberrypi kernel: [3815265.633521] [<8010ff30>] (unwind_backtrace) from [<8010c174>] (show_stack+0x20/0x24) Jan 28 23:48:32 raspberrypi kernel: [3815265.633531] [<8010c174>] (show_stack) from [<8078b424>] (dump_stack+0xd4/0x118) Jan 28 23:48:32 raspberrypi kernel: [3815265.633543] [<8078b424>] (dump_stack) from [<80224bac>] (dump_header+0xac/0x208) Jan 28 23:48:32 raspberrypi kernel: [3815265.633553] [<80224bac>] (dump_header) from [<80223f14>] (oom_kill_process+0x478/0x584) Jan 28 23:48:32 raspberrypi kernel: [3815265.633560] [<80223f14>] (oom_kill_process) from [<80224874>] (out_of_memory+0x124/0x334) Jan 28 23:48:32 raspberrypi kernel: [3815265.633570] [<80224874>] (out_of_memory) from [<8022a3b8>] (__alloc_pages_nodemask+0x10cc/0x11c0) Jan 28 23:48:32 raspberrypi kernel: [3815265.633579] [<8022a3b8>] (__alloc_pages_nodemask) from [<80275a60>] (new_slab+0x454/0x558) Jan 28 23:48:32 raspberrypi kernel: [3815265.633587] [<80275a60>] (new_slab) from [<802778a4>] (___slab_alloc.constprop.11+0x228/0x2c0) Jan 28 23:48:32 raspberrypi kernel: [3815265.633595] [<802778a4>] (___slab_alloc.constprop.11) from [<80277980>] (__slab_alloc.constprop.10+0x44/0x90) Jan 28 23:48:32 raspberrypi kernel: [3815265.633602] [<80277980>] (__slab_alloc.constprop.10) from [<80278118>] (kmem_cache_alloc+0x1f4/0x230) Jan 28 23:48:32 raspberrypi kernel: [3815265.633611] [<80278118>] (kmem_cache_alloc) from [<802f764c>] (proc_alloc_inode+0x2c/0x5c) Jan 28 23:48:32 raspberrypi kernel: [3815265.633622] [<802f764c>] (proc_alloc_inode) from [<802a7ca8>] (alloc_inode+0x2c/0xb4) Jan 28 23:48:32 raspberrypi kernel: [3815265.633631] [<802a7ca8>] (alloc_inode) from [<802aa01c>] (new_inode_pseudo+0x18/0x5c) Jan 28 23:48:32 raspberrypi kernel: [3815265.633639] [<802aa01c>] (new_inode_pseudo) from [<802aa07c>] (new_inode+0x1c/0x30) Jan 28 23:48:32 raspberrypi kernel: [3815265.633646] [<802aa07c>] (new_inode) from [<802fb850>] (proc_pid_make_inode+0x24/0xc0) Jan 28 23:48:32 raspberrypi kernel: [3815265.633654] [<802fb850>] (proc_pid_make_inode) from [<802fbdc8>] (proc_pident_instantiate+0x2c/0xb0) Jan 28 23:48:32 raspberrypi kernel: [3815265.633660] [<802fbdc8>] (proc_pident_instantiate) from [<802fbee8>] (proc_pident_lookup+0x9c/0xf0) Jan 28 23:48:32 raspberrypi kernel: [3815265.633667] [<802fbee8>] (proc_pident_lookup) from [<802fbf84>] (proc_tgid_base_lookup+0x20/0x28) Jan 28 23:48:32 raspberrypi kernel: [3815265.633675] [<802fbf84>] (proc_tgid_base_lookup) from [<8029ad60>] (path_openat+0xe0c/0x10c0) Jan 28 23:48:32 raspberrypi kernel: [3815265.633684] [<8029ad60>] (pat
[weewx-user] Re: Second crash after 11 days
Andrew and Rich...are you running the version of Stretch with the desktop, of just the Lite version? My system did crash last evening as I thought it would. Steve -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[weewx-user] Re: Second crash after 11 days
I'm running the same as you and not having any memory issues. pi@stormRPI3:~ $ cat /etc/os-release PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)" NAME="Raspbian GNU/Linux" VERSION_ID="9" VERSION="9 (stretch)" ID=raspbian ID_LIKE=debian HOME_URL="http://www.raspbian.org/"; SUPPORT_URL="http://www.raspbian.org/RaspbianForums"; BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"; pi@stormRPI3:~ $ cat /etc/debian_version 9.6 On Monday, January 28, 2019 at 10:30:54 PM UTC-5, Steve2Q wrote: > > Vince, I am running Raspian with the "image with Desktop base on Debian > Stretch" (this is from the download section of raspberrypi.org). Wonder > if it is possible that the image I used has some elements for running the > Desktop that may be causing the problem. I am just speculating and it would > be interesting to know if those who do not have this problem are either > using an "old" Raspian version or the newer Stretch Lite. > > Here are two reports from the command prompt: > > > > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[weewx-user] Re: Second crash after 11 days
Looks the same as mine!! pi@RPi3:~/perl $ cat /etc/os-release PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)" NAME="Raspbian GNU/Linux" VERSION_ID="9" VERSION="9 (stretch)" ID=raspbian ID_LIKE=debian HOME_URL="http://www.raspbian.org/"; SUPPORT_URL="http://www.raspbian.org/RaspbianForums"; BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"; pi@RPi3:~/perl $ cat /etc/debian_version 9.6 On Tuesday, 29 January 2019 05:30:54 UTC+2, Steve2Q wrote: > > Vince, I am running Raspian with the "image with Desktop base on Debian > Stretch" (this is from the download section of raspberrypi.org). Wonder > if it is possible that the image I used has some elements for running the > Desktop that may be causing the problem. I am just speculating and it would > be interesting to know if those who do not have this problem are either > using an "old" Raspian version or the newer Stretch Lite. > > Here are two reports from the command prompt: > > > > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[weewx-user] Re: Second crash after 11 days
Vince, I am running Raspian with the "image with Desktop base on Debian Stretch" (this is from the download section of raspberrypi.org). Wonder if it is possible that the image I used has some elements for running the Desktop that may be causing the problem. I am just speculating and it would be interesting to know if those who do not have this problem are either using an "old" Raspian version or the newer Stretch Lite. Here are two reports from the command prompt: -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[weewx-user] Re: Second crash after 11 days
On Monday, January 28, 2019 at 6:42:30 PM UTC-8, Steve2Q wrote: > > Vince; I used setup.py for installation. Additional info: Running Debian > 9.6 (Stretch) on a pi3 B+ . I do not have the "lite" version as I was > using the pi for other things. Do you think not having Stretch Lite could > be part of the problem? > > > Are you running Debian or Raspbian ? If you're running Debian, I can't speculate what the heck is in there. If you're running Raspbian, then there's nothing in it that should prevent things from working that I'm aware of. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[weewx-user] Re: Second crash after 11 days
Vince; I used setup.py for installation. Additional info: Running Debian 9.6 (Stretch) on a pi3 B+ . I do not have the "lite" version as I was using the pi for other things. Do you think not having Stretch Lite could be part of the problem? Steve -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[weewx-user] Re: Second crash after 11 days
On Monday, January 28, 2019 at 2:28:04 PM UTC-8, Steve2Q wrote: > > Gary..Yes, RTG is not enabled. At the time I am writing this, top yields > weewxd using 92.5% of memory. Putty is now very slow when trying to access > the pi, so I think it is very close to going down. The current up time is > 21D 23H 31M. I am going to let it run till it goes down. > > > I continue to be at a loss. Dozens and dozens of people are running on pi without experiencing this. There has to be something you've installed that is leaking. Usual suspect is imaging libraries. I'll install a clean pi3 tonight and let it run with the simulator without adding anything, using the current Raspbian Lite. Are you setup.py or apt-get for your installation method ? -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[weewx-user] Re: Second crash after 11 days
Gary..Yes, RTG is not enabled. At the time I am writing this, top yields weewxd using 92.5% of memory. Putty is now very slow when trying to access the pi, so I think it is very close to going down. The current up time is 21D 23H 31M. I am going to let it run till it goes down. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[weewx-user] Re: Second crash after 11 days
Steve, so just to be absolutely 100% clear, rtgd has not been running at any time while the attached graph was compiled? If that is the case there must be something fundamental causing the leak, I don't see a daily cron such as you describe causing this. I have not seen a plain vanilla WeeWX install do this. I have no ideas that are anything other than just clutching at straws. Gary On Monday, 28 January 2019 01:39:22 UTC+10, Steve2Q wrote: > > Ok..here is what has been happening so far. I am running 3.8.2 which is > "plain vanilla" with the exception of a cron which zips weewx.sdb once/day, > uploads the zip file to my website for storage, and send me an email that > the cron finished successfully. I thought the memory problem was gone > (probably associated with my RealTimeGauges). I have not had a crash, but > attached is a graph of the what the memory used by weexd looks like over > the past 22 days. It looks like it is going to crash in the next day or so. > Any more ideas? > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[weewx-user] Re: Second crash after 11 days
Ok..here is what has been happening so far. I am running 3.8.2 which is "plain vanilla" with the exception of a cron which zips weewx.sdb once/day, uploads the zip file to my website for storage, and send me an email that the cron finished successfully. I thought the memory problem was gone (probably associated with my RealTimeGauges). I have not had a crash, but attached is a graph of the what the memory used by weexd looks like over the past 22 days. It looks like it is going to crash in the next day or so. Any more ideas? -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
