[weewx-user] Re: NEW!!! --- OutOfSpan: Attempt to merge an accumulator whose timespan is not a subset

[Bryan Peter]

Matthew,

 Sorry forgot to answer your question on wx_ftp_upload and wx_cp_index. 
They are a couple of services I added in the early days of using weewx
wx_ftp_load - to upload some camera images from my local directories. I 
could easily remove and put the images in the html path, but just haven't 
got around to it.
wx_cp_index -  I was using as a cheap copy of the index.html file to a 
network drive so I can have a monitoring system check to see if the weewx 
system has failed and perform some restart operations and send out some 
admin email messages. One of my remote sites uses a couple of cheap 
TEMPerHum sensors that I have had  trouble with causing USB hangs on the 
RPi, so monitoring for a stale index file was the approach I took to detect 
this type of system failure. As you know USB hangs are problematic for 
Acurite stations as sometimes the Acurite stations stop talking completely 
when it happens. Only solution I have found that works consistently is to 
power down the weewx RPi and wait for 30mins for the Acurite to reset 
itself due to no USB traffic then reboot the RPi. So I set up a separate 
RPi as a system monitor and it handles all that via GPIO pins on the weewx 
RPi

As for stale data I have just started seeing that from both stations that 
past few weeks, generally not an issue. My thinking is that the batteries 
in the 5 in 1's are reaching the end of the life cycle and will need to be 
changed.

Thanks,

On Tuesday, February 5, 2019 at 9:22:37 AM UTC-5, mwall wrote:
>
> the OutOfSpan failure killed the report thread, and that brought down 
> weewx.  imho this is a bug - weewx should continue to collect data, even if 
> the report thread cannot do its job.  however, tom and gary know the 
> accumulator code better than i do, so they could say whether OutOfSpan 
> should be a fatal failure, versus a just "restart the report engine" 
> failure.  but first we need to know the root cause of OutOfSpan.
>
> a few observations:
>
> - here is the analysis of report generation times from the syslog.2 file.  
> nothing unusual here.
>
> report cycle for record 2019-02-01 06:28:00
> StdReport - 14 files in 18.82s + 36 images in 18.89s
> cmon - 1 file in 0.7s + 32 images in 82.48s
> exfoliation - 9 files in 100.62s + 62 images in 38.07s
> forecast - 12 files in 110.11s
> ftp - 353 files in 110.13s
>
> report cycle for record 2019-02-01 03:35:00
> skipped because report generator still running previous record
>
> report cycle for record 2019-02-01 06:42:00
> StdReport - 14 files in 5.53s + 12 images in 3.07s
> cmon - 1 file in 0.15s + 32 images in 81.32s
> exfoliation - 8 files in 9.64s + 48 images in 27.97s
> forecast - 12 files in 82.3s
> ftp - 127 files in 35.62s
>
> report cycle for record 2019-02-01 06:49:00
> StdReport - 14 files in 2.89s + 12 images in 2.89s
> cmon - 1 file in 0.15s + 32 images in 81.12s
> exfoliation - 8 files in 9.63s + 28 images in 9.37s
> forecast 12 files in 83.02s
> ftp - 107 files in 33.38s
>
> so the first run is longer, as expected.  you should see longer report 
> generation times whenever the monthly/yearly plots are updated as well.
>
> you could eliminate LOTS of overhead by removing the forecast skin.  the 
> exfoliation skin has forecast reporting built in, so you don't need the 
> forecast skin too.  the forecast skin is designed to illustrate how 
> forecasting works and to provide lots of examples of how to use it.  you 
> probably don't want to run it in 'production'.
>
> - what are wx_ftp_upload and wx_cp_index?
>
> - you are getting stale data from the acurite sensor cluster (see circa 
> 07:13:47 in syslog.2). this is not unusual, but if it happens a lot then 
> you should look at way of making your connection stronger (eliminate 
> interference, re-orient the console, put console closer to sensors, dance 
> naked around a fire at the next full moon, etc).
>
> - you might want to plot the 'ignoring stale data' messages over time.  
> that is a good way to see if there is a pattern to the interference timing.
>
> - don't generate forecasts any more often than you must.  once every 4 
> hours is typically sufficient.  i think exfoliation is set up to do this - 
> it only generates forecast.html every 4 hours.  and the forecast download 
> services are set up to download the data based on when the sources update.  
> once again, get rid of forecast skin, because i think it regenerates every 
> report cycle, even though that frequency is pointless.
>
> - be sure that you have a sane max_age in the [Forecast] section in 
> weewx.conf.  something like 604800 is reasonable.  you should only use 
> max_age=none if you intend to retain forecasts indefinitely, e.g., for 
> doing forecast comparisons over extended periods.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more opt

Re: [weewx-user] Re: NEW!!! --- OutOfSpan: Attempt to merge an accumulator whose timespan is not a subset

[Bryan Peter]

Tom,

It seems that with Acurite no matter what setting is for record 
generation it is really always using software. I have run a couple of tests 
with different archive_interval settings that indicate there is some kind 
of issue with the accumulator (accum.py) code that triggers this shutdown. 
If I use an archive_interval value that is integer divisible into the 
1440mins, I used 360 for my test, of a 1 day cycle the timestamp for 
weewx.sdb record additions matches to the minute with the system time stamp 
in the syslog log. However, if I use the archive_interval of 7min (420), or 
I suspect 14min (840) the timestamps for weewx.sdb get off track pretty 
quickly (the issue that Gary pointed out).

As you say this is likely a small issue, but may make sense at some point 
to not allow certain archive intervals given that they cause issues for 
accumulated stat generation processes.

As an aside, why does cmon skin always have the correct and exact match to 
the system timestamp for its record add time? I'm assuming it is because it 
does not need to do accumulated stats so it just grabs the system time 
instead of calculating time as accum.py, manager.py, and engine.py do. 

Thanks,

Bryan

On Tuesday, February 5, 2019 at 9:43:58 AM UTC-5, Thomas Keffer wrote:
>
> On Tue, Feb 5, 2019 at 6:22 AM mwall > 
> wrote:
>
>> the OutOfSpan failure killed the report thread, and that brought down 
>> weewx.  imho this is a bug - weewx should continue to collect data, even if 
>> the report thread cannot do its job. 
>>
>
> I'm not seeing that. When the engine shuts down, it calls the shutDown() 
> method of all running services. Then the engine exits. I think what you are 
> seeing is the results of calling StdReport.shutDown().
>
> Perhaps this could be made more clear by having the engine put a message 
> in the log to the effect ("Engine exiting; starting shutdown of all 
> services") before calling all the shutDown() methods.
>
> As to the root cause, it's probably the unusual archive interval (420 
> seconds). There may be an underlying assumption in the code that the 
> midnight boundary will always fall at the end of an archive interval and, 
> when it doesn't, this failure occurs. I'd have to test.
>
> But, is it important? Yes, there's probably a bug, but it's a pretty minor 
> one. Alternatively, we could test the archive interval to make sure it's 
> divisible into 60 and refuse to run if it isn't.
>
> -tk
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: NEW!!! --- OutOfSpan: Attempt to merge an accumulator whose timespan is not a subset

[Bryan Peter]

Matthew,

 Thanks for the helpful tips will implement to reduce overhead. It runs 
fairly sanely with an archive interval of 360 with forecast skin in so 
taking that out should get me back to 300.

Best,

Bryan


On Tuesday, February 5, 2019 at 9:22:37 AM UTC-5, mwall wrote:
>
> the OutOfSpan failure killed the report thread, and that brought down 
> weewx.  imho this is a bug - weewx should continue to collect data, even if 
> the report thread cannot do its job.  however, tom and gary know the 
> accumulator code better than i do, so they could say whether OutOfSpan 
> should be a fatal failure, versus a just "restart the report engine" 
> failure.  but first we need to know the root cause of OutOfSpan.
>
> a few observations:
>
> - here is the analysis of report generation times from the syslog.2 file.  
> nothing unusual here.
>
> report cycle for record 2019-02-01 06:28:00
> StdReport - 14 files in 18.82s + 36 images in 18.89s
> cmon - 1 file in 0.7s + 32 images in 82.48s
> exfoliation - 9 files in 100.62s + 62 images in 38.07s
> forecast - 12 files in 110.11s
> ftp - 353 files in 110.13s
>
> report cycle for record 2019-02-01 03:35:00
> skipped because report generator still running previous record
>
> report cycle for record 2019-02-01 06:42:00
> StdReport - 14 files in 5.53s + 12 images in 3.07s
> cmon - 1 file in 0.15s + 32 images in 81.32s
> exfoliation - 8 files in 9.64s + 48 images in 27.97s
> forecast - 12 files in 82.3s
> ftp - 127 files in 35.62s
>
> report cycle for record 2019-02-01 06:49:00
> StdReport - 14 files in 2.89s + 12 images in 2.89s
> cmon - 1 file in 0.15s + 32 images in 81.12s
> exfoliation - 8 files in 9.63s + 28 images in 9.37s
> forecast 12 files in 83.02s
> ftp - 107 files in 33.38s
>
> so the first run is longer, as expected.  you should see longer report 
> generation times whenever the monthly/yearly plots are updated as well.
>
> you could eliminate LOTS of overhead by removing the forecast skin.  the 
> exfoliation skin has forecast reporting built in, so you don't need the 
> forecast skin too.  the forecast skin is designed to illustrate how 
> forecasting works and to provide lots of examples of how to use it.  you 
> probably don't want to run it in 'production'.
>
> - what are wx_ftp_upload and wx_cp_index?
>
> - you are getting stale data from the acurite sensor cluster (see circa 
> 07:13:47 in syslog.2). this is not unusual, but if it happens a lot then 
> you should look at way of making your connection stronger (eliminate 
> interference, re-orient the console, put console closer to sensors, dance 
> naked around a fire at the next full moon, etc).
>
> - you might want to plot the 'ignoring stale data' messages over time.  
> that is a good way to see if there is a pattern to the interference timing.
>
> - don't generate forecasts any more often than you must.  once every 4 
> hours is typically sufficient.  i think exfoliation is set up to do this - 
> it only generates forecast.html every 4 hours.  and the forecast download 
> services are set up to download the data based on when the sources update.  
> once again, get rid of forecast skin, because i think it regenerates every 
> report cycle, even though that frequency is pointless.
>
> - be sure that you have a sane max_age in the [Forecast] section in 
> weewx.conf.  something like 604800 is reasonable.  you should only use 
> max_age=none if you intend to retain forecasts indefinitely, e.g., for 
> doing forecast comparisons over extended periods.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: NEW!!! --- OutOfSpan: Attempt to merge an accumulator whose timespan is not a subset

[Thomas Keffer]

On Tue, Feb 5, 2019 at 6:22 AM mwall  wrote:

> the OutOfSpan failure killed the report thread, and that brought down
> weewx.  imho this is a bug - weewx should continue to collect data, even if
> the report thread cannot do its job.
>

I'm not seeing that. When the engine shuts down, it calls the shutDown()
method of all running services. Then the engine exits. I think what you are
seeing is the results of calling StdReport.shutDown().

Perhaps this could be made more clear by having the engine put a message in
the log to the effect ("Engine exiting; starting shutdown of all services")
before calling all the shutDown() methods.

As to the root cause, it's probably the unusual archive interval (420
seconds). There may be an underlying assumption in the code that the
midnight boundary will always fall at the end of an archive interval and,
when it doesn't, this failure occurs. I'd have to test.

But, is it important? Yes, there's probably a bug, but it's a pretty minor
one. Alternatively, we could test the archive interval to make sure it's
divisible into 60 and refuse to run if it isn't.

-tk

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: NEW!!! --- OutOfSpan: Attempt to merge an accumulator whose timespan is not a subset

[mwall]

the OutOfSpan failure killed the report thread, and that brought down 
weewx.  imho this is a bug - weewx should continue to collect data, even if 
the report thread cannot do its job.  however, tom and gary know the 
accumulator code better than i do, so they could say whether OutOfSpan 
should be a fatal failure, versus a just "restart the report engine" 
failure.  but first we need to know the root cause of OutOfSpan.

a few observations:

- here is the analysis of report generation times from the syslog.2 file.  
nothing unusual here.

report cycle for record 2019-02-01 06:28:00
StdReport - 14 files in 18.82s + 36 images in 18.89s
cmon - 1 file in 0.7s + 32 images in 82.48s
exfoliation - 9 files in 100.62s + 62 images in 38.07s
forecast - 12 files in 110.11s
ftp - 353 files in 110.13s

report cycle for record 2019-02-01 03:35:00
skipped because report generator still running previous record

report cycle for record 2019-02-01 06:42:00
StdReport - 14 files in 5.53s + 12 images in 3.07s
cmon - 1 file in 0.15s + 32 images in 81.32s
exfoliation - 8 files in 9.64s + 48 images in 27.97s
forecast - 12 files in 82.3s
ftp - 127 files in 35.62s

report cycle for record 2019-02-01 06:49:00
StdReport - 14 files in 2.89s + 12 images in 2.89s
cmon - 1 file in 0.15s + 32 images in 81.12s
exfoliation - 8 files in 9.63s + 28 images in 9.37s
forecast 12 files in 83.02s
ftp - 107 files in 33.38s

so the first run is longer, as expected.  you should see longer report 
generation times whenever the monthly/yearly plots are updated as well.

you could eliminate LOTS of overhead by removing the forecast skin.  the 
exfoliation skin has forecast reporting built in, so you don't need the 
forecast skin too.  the forecast skin is designed to illustrate how 
forecasting works and to provide lots of examples of how to use it.  you 
probably don't want to run it in 'production'.

- what are wx_ftp_upload and wx_cp_index?

- you are getting stale data from the acurite sensor cluster (see circa 
07:13:47 in syslog.2). this is not unusual, but if it happens a lot then 
you should look at way of making your connection stronger (eliminate 
interference, re-orient the console, put console closer to sensors, dance 
naked around a fire at the next full moon, etc).

- you might want to plot the 'ignoring stale data' messages over time.  
that is a good way to see if there is a pattern to the interference timing.

- don't generate forecasts any more often than you must.  once every 4 
hours is typically sufficient.  i think exfoliation is set up to do this - 
it only generates forecast.html every 4 hours.  and the forecast download 
services are set up to download the data based on when the sources update.  
once again, get rid of forecast skin, because i think it regenerates every 
report cycle, even though that frequency is pointless.

- be sure that you have a sane max_age in the [Forecast] section in 
weewx.conf.  something like 604800 is reasonable.  you should only use 
max_age=none if you intend to retain forecasts indefinitely, e.g., for 
doing forecast comparisons over extended periods.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: NEW!!! --- OutOfSpan: Attempt to merge an accumulator whose timespan is not a subset

[gjr80]

Hi,

This gives us a clue to what is going on and the possible cause of the 
OutOfSpan error :

Feb  2 07:19:17 bdp-rpiz1 weewx[21597]: manager: Added record 2019-02-02 07:
19:17 EST (1549109957) to database 'cmon.sdb'
Feb  2 07:19:17 bdp-rpiz1 weewx[21597]: manager: Added record 2019-02-02 07:
19:17 EST (1549109957) to daily summary in 'cmon.sdb'
Feb  2 07:19:18 bdp-rpiz1 weewx[21597]: manager: Added record 2019-02-02 07:
14:00 EST (1549109640) to database 'weewx.sdb'
Feb  2 07:19:18 bdp-rpiz1 weewx[21597]: manager: Added record 2019-02-02 07:
14:00 EST (1549109640) to daily summary in 'weewx.sdb'

I suspect that the large difference between the actual system time and the 
timestamp of the archive record being saved to weewx.sdb is upsetting WeeWX 
when the accumulators change at midnight. The WeeWX accumulators work on a 
midnight-to-midnight period and the recipt of the first loop data 
timestamped after midnight causes a new accumulator to be created and used. 
Once WeeWX has moved on to a new accumulator for the current day it does 
not like to see an old data from the previous day. Restarting WeeWX may 
deal with the fatal error but it will certainly not cure it, it will 
continue to occur shortly after midnight.

You might try to edit weewx.conf and under [StdArchive] set record_generation 
= software, the Acurite driver does not support downloading archive records 
from the station and WeeWX should fall back to software record generation 
but let's force it right from the start. I would also put the archive 
interval back to 300, having it at 420 will do nothing to help with the 
issue, rather the odd interval only serves to make interpreting the logs 
more difficult. [StdArchive] should look something like (changes 
highlighted):

[StdArchive]

# If the station hardware supports data logging then the archive 
interval
# will be downloaded from the station. Otherwise, specify it (in 
seconds).
archive_interval = 300

# If possible, new archive records are downloaded from the station
# hardware. If the hardware does not support this, then new archive
# records will be generated in software.
# Set the following to "software" to force software record generation.
record_generation = software


After making these changes save weewx.conf then restart WeeWX. Monitor the 
logs to see what happens around midnight.

If this does not fix the problem could you run WeeWX directly 
. This will display 
on the console (screen) the loop data from your station (indicated by lines 
starting with LOOP:) as well as the archive records generated by WeeWX 
(indicated by lines starting with (REC:). Let this run for 10-15 minutes 
and post a capture of the data displayed on the console.

Gary

On Monday, 4 February 2019 03:06:57 UTC+10, Bryan Peter wrote:
>
> I have been seeing the following Error/Exit of weewx over the past several 
> days. Here are the details of my configuration:
>
> Wx Station: Acurite 01525 with 5 in 1 sensor
> Server: Rpi Zero W with attached powered USB 3.0 Hub with attached hard 
> drive for /var and /home
> OS: Linux bdp-rpiz1 4.14.79+ #1159 Sun Nov 4 17:28:08 GMT 2018 armv6l 
> GNU/Linux
> weewx version: 3.8.2
> extensions: cmon 0.16, exfoliation 0.45
>
> Installation info and details:
>
> I have been running weewx for a few years successfully on various versions 
> of Rpi and Intel systems using Acurite stations
> This install was a fresh install on the Pi Zero W. I had this station 
> running on an Intel box that suddenly lost its Wifi so decided to 
> transition to the Zero as a lower energy replacement.
> ran the simulator station type for several days to make sure install 
> working and found no issues. Made cut over from Intel to Pi zero by doing 
> the following:
>
> 1) stop weewx on RpiZ
> 2) wee_config  (to add the Acurite driver to my config file)
> 3) deleted the old simulator mode database files from /var/lib
> 4) stop weewx on Intel box
> 5) wee_database --drop-daily  (on Intel box)
> 6) wee_database --rebuild-daily (on Intel box)
> 7) copied cmon.sdb, forecast.sdb, weewx.sdb form Intel box to RpiZ
> 8) restart weewx on RpiZ
>
> Monitored logs for a time and things looked good except I was seeing that 
> the RpiZ was having trouble keeping up with Image generation, forecast file 
> generation cycles taking 4X longer than on my Rpi3. So ran into some 
> StdReport Thread aborts due to current report still running. 
> I let this go for the first day or so things seemed fine then I saw the 
> first OutOfSpan error log with weewx halting during the overnight (just 
> after midnight). It was very strange though as the system did not show 
> weewx running in the process status table or systemctl status data, but 
> attempts to restart weewx via "sudo service weewx start", did not work. I 
> found I had to issue "sudo service weewx stop" to clean some system status 
> up to restart weewx.