[weewx-user] Re: WeeWX crashing for unknown reasons

[rkr...@gmail.com]

Nevermind, stared at it long enough that I solved my own problem!   Never, 
ever trust external data!  I pull data from the US Bureau of Reclamation 
(lake elevation, dam opening, discharge, etc..)  I had assumed their data 
was good, and wasn't doing the proper checks in my import script.  Turns 
out they are having issues, and the dam elevation now randomly has the 
wrong value &/or a minus sign.  WeeWX didn't like being fed something with 
a minus sign at the end, where it expects a float.

On Thursday, May 5, 2022 at 8:03:15 AM UTC-5 rkr...@gmail.com wrote:

> I've had my station (laketschida.info) up for 2+ years, and recently it 
> developed a new bad habit.  WeeWX randomly decides to crash with very 
> little useful error messages.  It just basically stops.  And when I restart 
> it, it often takes a substantial length of time (hours) before it actually 
> starts updating the reports, or even writing to the archive.sdb.  (the mqqt 
> feed works fine during this time)
>
> Any ideas?  Debug and /var/log/messages attached
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/553120da-776a-47c3-bed2-c3549c2880e2n%40googlegroups.com.

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[Redanman]

Thank you.  Problem solved (hopefully).
Mike

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/405a6317-7bc8-40a5-9970-5d33d08e837c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[Luc Heijst]

On Tuesday, 7 May 2019 03:50:13 UTC-3, Redanman wrote:
>
> My system configuration is :
> cmon version = 0.16
>

Hi Redanman,

This is a known issue, see: 
https://groups.google.com/forum/#!searchin/weewx-user/cmon%7Csort:date/weewx-user/p8xhPGK38s4/a3HAj6eFCQAJ

For 64-bit systems the net stats values will have type bigint in python2 
when they have reached their max values.
Note: Even when the net stats value is in the range of 0..maxint, the type 
will still be bigint in such a case.
See my solutions in the tread mentioned.

Luc 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/3ea181a1-f4c9-4851-b6e0-629cb5eeb9ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[Redanman]

OK, that's done and I've checked that the cmon parameters are being written 
to the syslog.  Let's see what happens.
Mike

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/bd737b0c-6f4f-40f9-8294-b81e16022adf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[gjr80]

OK, as far as I can tell cmon 0.16 is doing what it should and it should 
work fine with a RPi/Stretch. Clearly though something is not working 
correctly. Cmon is mildly complex with not a lot of debug output so in the 
absence of Matthew's input could you try making a change to the cmon code. 
First up make a copy of cmon.py just in case:

$ sudo usr/share/weewx/user/cmon.py usr/share/weewx/user/cmon_orig.py

Then edit usr/share/weewx/user/cmon.py and around line 711 change:

def save_data(self, record):
"""save data to database"""
self.dbm.addRecord(record)

to:

def save_data(self, record):
"""save data to database"""
loginf("cmon archive record=%s" % (record,))
self.dbm.addRecord(record)

Save cmon.py and restart WeeWX. Cmon will now log the archive record it is 
about to save. Next time cmon causes a crash post the log and we should be 
able to marry up the value in the stack trace with a field in the cmon 
archive record.

Gary

On Tuesday, 7 May 2019 16:50:13 UTC+10, Redanman wrote:
>
> My system configuration is :
>
> Weewx version 3.9.1 running on a Raspberry Pi Zero
> Aercus Weathersleuth weather station with Weewx using Interceptor driver 
> over local network
> PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
> NAME="Raspbian GNU/Linux"
> VERSION_ID="9"
> VERSION="9 (stretch)"
> ID=raspbian
> ID_LIKE=debian
> Kernel = 4.14.62+ #1134 Tue Aug 14 16:58:07 BST 2018 armv6l GNU/Linux
> cmon version = 0.16
>
> Hope that helps.
> Mike
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/878be265-1d20-4959-b499-c590b769ed52%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[Redanman]

My system configuration is :

Weewx version 3.9.1 running on a Raspberry Pi Zero
Aercus Weathersleuth weather station with Weewx using Interceptor driver 
over local network
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
Kernel = 4.14.62+ #1134 Tue Aug 14 16:58:07 BST 2018 armv6l GNU/Linux
cmon version = 0.16

Hope that helps.
Mike

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/23f3e30d-5716-4edc-ad03-1842fb5247c3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[Thomas Keffer]

Or, cmon is reading a value from a /proc file, which are generally strings.
Most likely, the author forgot to convert to a float.

On Mon, May 6, 2019 at 4:05 PM gjr80  wrote:

> So the error is arising from data coming from cmon and not the weather
> station:
>
> May  6 20:20:31 raspberrypi weewx[333]: File
> "/usr/share/weewx/user/cmon.py", line 711, in save_data
> May  6 20:20:31 raspberrypi weewx[333]:   self.dbm.addRecord(
> record)
>
> and
>
> Apr 25 19:00:32 raspberrypi weewx[331]:  File
> "/usr/share/weewx/user/cmon.py", line 704, in new_archive_record
> Apr 25 19:00:32 raspberrypi weewx[331]:  self.save_data(self.get_data(
> now, self.last_ts))
> Apr 25 19:00:32 raspberrypi weewx[331]:  File
> "/usr/share/weewx/user/cmon.py", line 711, in save_data
> Apr 25 19:00:32 raspberrypi weewx[331]:  self.dbm.addRecord(record)
>
>
> Judging by the 'numbers' involved I would hazard a guess that the routine 
> used by cmon to extract data from some files on the system is returning 
> something other than a float or an int. Judging by the numbers I would be 
> looking at the network traffic.
> I recall an earlier thread where network traffic figures being read by cmon 
> were too big for an int, though that would not appear to be the case here.
>
> Perhaps some details of what cmon version, what operating system and WeeWX 
> system hardware might help
>
> Gary
>
>
> On Tuesday, 7 May 2019 07:56:45 UTC+10, Redanman wrote:
>>
>> Weewx crashed again tonight with what seems to be the same root cause.
>> Here is the syslog just at the moment it crashed.  Again, looking at the
>> data, I cannot see any spurious text/numbers.  Out of interest, is accum.py
>> run every iteration of the engine or only on an interval?
>>
>> May  6 20:20:30 raspberrypi weewx[333]: interceptor: MainThread: raw
>> packet: {'wind_speed': 0.0, 'humidity_in': 43.0, 'temperature_in': 68.4,
>> 'barometer': 30.15, 'windchill': 46.9, 'dewpoint': 40.5, 'battery': 0.0,
>> 'humidity_out': 78.0, 'uv': 0.0, 'radiation': 0.15, 'rain': 0.0,
>> 'dateTime': 1557170426, 'temperature_out': 46.9, 'wind_dir': 143.0,
>> 'rain_total': 4.49, 'usUnits': 1, 'wind_gust': 0.0}
>> May  6 20:20:30 raspberrypi weewx[333]: interceptor: MainThread: mapped
>> packet: {'barometer': 30.15, 'windchill': 46.9, 'dewpoint': 40.5,
>> 'outHumidity': 78.0, 'UV': 0.0, 'radiation': 0.15, 'rain': 0.0, 'dateTime':
>> 1557170426, 'windDir': 143.0, 'outTemp': 46.9, 'windSpeed': 0.0,
>> 'txBatteryStatus': 0.0, 'inTemp': 68.4, 'windGust': 0.0, 'inHumidity':
>> 43.0, 'usUnits': 1}
>> May  6 20:20:30 raspberrypi weewx[333]: manager: Added record 2019-05-06
>> 20:20:31 BST (1557170431) to database 'cmon.sdb'
>> May  6 20:20:30 raspberrypi weewx[333]: engine: Main loop exiting.
>> Shutting engine down.
>> May  6 20:20:30 raspberrypi weewx[333]: engine: Shutting down StdReport
>> thread
>> May  6 20:20:30 raspberrypi weewx[333]: engine: StdReport thread has been
>> terminated
>> May  6 20:20:30 raspberrypi weewx[333]: restx: Shut down WOW thread.
>> May  6 20:20:30 raspberrypi weewx[333]: restx: Shut down PWSWeather
>> thread.
>> May  6 20:20:31 raspberrypi weewx[333]: restx: Shut down Wunderground-PWS
>> thread.
>> May  6 20:20:31 raspberrypi weewx[333]: restx: Shut down StationRegistry
>> thread.
>> May  6 20:20:31 raspberrypi weewx[333]: interceptor: MainThread: shutting
>> down server thread
>> May  6 20:20:31 raspberrypi weewx[333]: interceptor: MainThread: stop tcp
>> server
>> May  6 20:20:31 raspberrypi weewx[333]: engine: Caught unrecoverable
>> exception in engine:
>> May  6 20:20:31 raspberrypi weewx[333]:   accum:
>> ScalarStats.addHiLo expected float or int, got 710464
>> May  6 20:20:31 raspberrypi weewx[333]:   Traceback (most recent
>> call last):
>> May  6 20:20:31 raspberrypi weewx[333]: File
>> "/usr/share/weewx/weewx/engine.py", line 890, in main
>> May  6 20:20:31 raspberrypi weewx[333]:   engine.run()
>> May  6 20:20:31 raspberrypi weewx[333]: File
>> "/usr/share/weewx/weewx/engine.py", line 202, in run
>> May  6 20:20:31 raspberrypi weewx[333]: 
>> self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
>> May  6 20:20:31 raspberrypi weewx[333]: File
>> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
>> May  6 20:20:31 raspberrypi weewx[333]:   callback(event)
>> May  6 20:20:31 raspberrypi weewx[333]: File
>> "/usr/share/weewx/weewx/engine.py", line 574, in post_loop
>> May  6 20:20:31 raspberrypi weewx[333]: 
>> self._software_catchup()
>> May  6 20:20:31 raspberrypi weewx[333]: File
>> "/usr/share/weewx/weewx/engine.py", line 646, in _software_catchup
>> May  6 20:20:31 raspberrypi weewx[333]: 
>> self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD,
>> record=record, origin='software'))
>> May  6 20:20:31 raspberrypi weewx[333]: File
>> "/usr/share/weewx/weewx/engine.py", line

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[gjr80]

So the error is arising from data coming from cmon and not the weather 
station:

May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/user/cmon.py", line 711, in save_data
May  6 20:20:31 raspberrypi weewx[333]:   self.dbm.addRecord(
record)

and

Apr 25 19:00:32 raspberrypi weewx[331]:  File 
"/usr/share/weewx/user/cmon.py", line 704, in new_archive_record
Apr 25 19:00:32 raspberrypi weewx[331]:  self.save_data(self.get_data(
now, self.last_ts))
Apr 25 19:00:32 raspberrypi weewx[331]:  File 
"/usr/share/weewx/user/cmon.py", line 711, in save_data
Apr 25 19:00:32 raspberrypi weewx[331]:  self.dbm.addRecord(record)


Judging by the 'numbers' involved I would hazard a guess that the routine used 
by cmon to extract data from some files on the system is returning something 
other than a float or an int. Judging by the numbers I would be looking at the 
network traffic. 
I recall an earlier thread where network traffic figures being read by cmon 
were too big for an int, though that would not appear to be the case here.

Perhaps some details of what cmon version, what operating system and WeeWX 
system hardware might help 

Gary


On Tuesday, 7 May 2019 07:56:45 UTC+10, Redanman wrote:
>
> Weewx crashed again tonight with what seems to be the same root cause.  
> Here is the syslog just at the moment it crashed.  Again, looking at the 
> data, I cannot see any spurious text/numbers.  Out of interest, is accum.py 
> run every iteration of the engine or only on an interval?  
>
> May  6 20:20:30 raspberrypi weewx[333]: interceptor: MainThread: raw 
> packet: {'wind_speed': 0.0, 'humidity_in': 43.0, 'temperature_in': 68.4, 
> 'barometer': 30.15, 'windchill': 46.9, 'dewpoint': 40.5, 'battery': 0.0, 
> 'humidity_out': 78.0, 'uv': 0.0, 'radiation': 0.15, 'rain': 0.0, 
> 'dateTime': 1557170426, 'temperature_out': 46.9, 'wind_dir': 143.0, 
> 'rain_total': 4.49, 'usUnits': 1, 'wind_gust': 0.0}
> May  6 20:20:30 raspberrypi weewx[333]: interceptor: MainThread: mapped 
> packet: {'barometer': 30.15, 'windchill': 46.9, 'dewpoint': 40.5, 
> 'outHumidity': 78.0, 'UV': 0.0, 'radiation': 0.15, 'rain': 0.0, 'dateTime': 
> 1557170426, 'windDir': 143.0, 'outTemp': 46.9, 'windSpeed': 0.0, 
> 'txBatteryStatus': 0.0, 'inTemp': 68.4, 'windGust': 0.0, 'inHumidity': 
> 43.0, 'usUnits': 1}
> May  6 20:20:30 raspberrypi weewx[333]: manager: Added record 2019-05-06 
> 20:20:31 BST (1557170431) to database 'cmon.sdb'
> May  6 20:20:30 raspberrypi weewx[333]: engine: Main loop exiting. 
> Shutting engine down.
> May  6 20:20:30 raspberrypi weewx[333]: engine: Shutting down StdReport 
> thread
> May  6 20:20:30 raspberrypi weewx[333]: engine: StdReport thread has been 
> terminated
> May  6 20:20:30 raspberrypi weewx[333]: restx: Shut down WOW thread.
> May  6 20:20:30 raspberrypi weewx[333]: restx: Shut down PWSWeather thread.
> May  6 20:20:31 raspberrypi weewx[333]: restx: Shut down Wunderground-PWS 
> thread.
> May  6 20:20:31 raspberrypi weewx[333]: restx: Shut down StationRegistry 
> thread.
> May  6 20:20:31 raspberrypi weewx[333]: interceptor: MainThread: shutting 
> down server thread
> May  6 20:20:31 raspberrypi weewx[333]: interceptor: MainThread: stop tcp 
> server
> May  6 20:20:31 raspberrypi weewx[333]: engine: Caught unrecoverable 
> exception in engine:
> May  6 20:20:31 raspberrypi weewx[333]:   accum: 
> ScalarStats.addHiLo expected float or int, got 710464
> May  6 20:20:31 raspberrypi weewx[333]:   Traceback (most recent 
> call last):
> May  6 20:20:31 raspberrypi weewx[333]: File 
> "/usr/share/weewx/weewx/engine.py", line 890, in main
> May  6 20:20:31 raspberrypi weewx[333]:   engine.run()
> May  6 20:20:31 raspberrypi weewx[333]: File 
> "/usr/share/weewx/weewx/engine.py", line 202, in run
> May  6 20:20:31 raspberrypi weewx[333]:   
> self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
> May  6 20:20:31 raspberrypi weewx[333]: File 
> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
> May  6 20:20:31 raspberrypi weewx[333]:   callback(event)
> May  6 20:20:31 raspberrypi weewx[333]: File 
> "/usr/share/weewx/weewx/engine.py", line 574, in post_loop
> May  6 20:20:31 raspberrypi weewx[333]:   
> self._software_catchup()
> May  6 20:20:31 raspberrypi weewx[333]: File 
> "/usr/share/weewx/weewx/engine.py", line 646, in _software_catchup
> May  6 20:20:31 raspberrypi weewx[333]:   
> self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD, 
> record=record, origin='software'))
> May  6 20:20:31 raspberrypi weewx[333]: File 
> "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
> May  6 20:20:31 raspberrypi weewx[333]:   callback(event)
> May  6 20:20:31 raspberrypi weewx[333]: File 
> "/usr/share/weewx/user/cmon.py", line 704, in new_archive_record
> May  6 20:20:31 ra

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[Redanman]

Weewx crashed again tonight with what seems to be the same root cause.  
Here is the syslog just at the moment it crashed.  Again, looking at the 
data, I cannot see any spurious text/numbers.  Out of interest, is accum.py 
run every iteration of the engine or only on an interval?  

May  6 20:20:30 raspberrypi weewx[333]: interceptor: MainThread: raw 
packet: {'wind_speed': 0.0, 'humidity_in': 43.0, 'temperature_in': 68.4, 
'barometer': 30.15, 'windchill': 46.9, 'dewpoint': 40.5, 'battery': 0.0, 
'humidity_out': 78.0, 'uv': 0.0, 'radiation': 0.15, 'rain': 0.0, 
'dateTime': 1557170426, 'temperature_out': 46.9, 'wind_dir': 143.0, 
'rain_total': 4.49, 'usUnits': 1, 'wind_gust': 0.0}
May  6 20:20:30 raspberrypi weewx[333]: interceptor: MainThread: mapped 
packet: {'barometer': 30.15, 'windchill': 46.9, 'dewpoint': 40.5, 
'outHumidity': 78.0, 'UV': 0.0, 'radiation': 0.15, 'rain': 0.0, 'dateTime': 
1557170426, 'windDir': 143.0, 'outTemp': 46.9, 'windSpeed': 0.0, 
'txBatteryStatus': 0.0, 'inTemp': 68.4, 'windGust': 0.0, 'inHumidity': 
43.0, 'usUnits': 1}
May  6 20:20:30 raspberrypi weewx[333]: manager: Added record 2019-05-06 
20:20:31 BST (1557170431) to database 'cmon.sdb'
May  6 20:20:30 raspberrypi weewx[333]: engine: Main loop exiting. Shutting 
engine down.
May  6 20:20:30 raspberrypi weewx[333]: engine: Shutting down StdReport 
thread
May  6 20:20:30 raspberrypi weewx[333]: engine: StdReport thread has been 
terminated
May  6 20:20:30 raspberrypi weewx[333]: restx: Shut down WOW thread.
May  6 20:20:30 raspberrypi weewx[333]: restx: Shut down PWSWeather thread.
May  6 20:20:31 raspberrypi weewx[333]: restx: Shut down Wunderground-PWS 
thread.
May  6 20:20:31 raspberrypi weewx[333]: restx: Shut down StationRegistry 
thread.
May  6 20:20:31 raspberrypi weewx[333]: interceptor: MainThread: shutting 
down server thread
May  6 20:20:31 raspberrypi weewx[333]: interceptor: MainThread: stop tcp 
server
May  6 20:20:31 raspberrypi weewx[333]: engine: Caught unrecoverable 
exception in engine:
May  6 20:20:31 raspberrypi weewx[333]:   accum: 
ScalarStats.addHiLo expected float or int, got 710464
May  6 20:20:31 raspberrypi weewx[333]:   Traceback (most recent 
call last):
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/weewx/engine.py", line 890, in main
May  6 20:20:31 raspberrypi weewx[333]:   engine.run()
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/weewx/engine.py", line 202, in run
May  6 20:20:31 raspberrypi weewx[333]:   
self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
May  6 20:20:31 raspberrypi weewx[333]:   callback(event)
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/weewx/engine.py", line 574, in post_loop
May  6 20:20:31 raspberrypi weewx[333]:   
self._software_catchup()
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/weewx/engine.py", line 646, in _software_catchup
May  6 20:20:31 raspberrypi weewx[333]:   
self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD, 
record=record, origin='software'))
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
May  6 20:20:31 raspberrypi weewx[333]:   callback(event)
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/user/cmon.py", line 704, in new_archive_record
May  6 20:20:31 raspberrypi weewx[333]:   
self.save_data(self.get_data(now, self.last_ts))
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/user/cmon.py", line 711, in save_data
May  6 20:20:31 raspberrypi weewx[333]:   
self.dbm.addRecord(record)
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/weewx/manager.py", line 246, in addRecord
May  6 20:20:31 raspberrypi weewx[333]:   
self._addSingleRecord(record, cursor, log_level)
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/weewx/manager.py", line 1216, in _addSingleRecord
May  6 20:20:31 raspberrypi weewx[333]:   
_day_summary.addRecord(record, weight=_weight)
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/weewx/accum.py", line 256, in addRecord
May  6 20:20:31 raspberrypi weewx[333]:   func(self, record, 
obs_type, add_hilo, weight)
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/weewx/accum.py", line 314, in add_value
May  6 20:20:31 raspberrypi weewx[333]:   
self[obs_type].addHiLo(val, record['dateTime'])
May  6 20:20:31 raspberrypi weewx[333]: File 
"/usr/share/weewx/weewx/accum.py", line 77, in addHiLo
May  6 20:20:31 raspberrypi weewx[333]:   raise 
ValueError("accum: ScalarStats.addHiLo expe

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[pliggen . is]

Den lördag 27 april 2019 kl. 02:14:15 UTC+2 skrev mwall:
>
> On Friday, April 26, 2019 at 6:34:50 PM UTC-4, pliggen.is wrote:
>>
>> I also got the issue two days ago. Strange because I've been running for 
>> quite some time now and just the other day was my first crash. 
>>
>
> what kind of weather station?
>
> what is your interceptor configuration?  (please post the [Interceptor] 
> section from your weewx configuration file)
>
> m
>

Not sure what interceptor means, maybe something in this?

[FineOffsetUSB]
# This section is for the Fine Offset series of weather stations.

# The station model, e.g., WH1080, WS1090, WS2080, WH3081
model = WH1080

# How often to poll the station for data, in seconds
polling_interval = 60

# The driver to use:
driver = weewx.drivers.fousb


-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[Redanman]

Sorry, negative on both counts.  I've looked back in the syslog files and I 
cannot see any problems with the parsing of the interceptor data, or any 
sign of the value 645138 or 667872.  The weewx.sdb file is full of numbers 
or null.  The other thing i can mention is that a restart of weewx did not 
clear the problem, it needed a sudo reboot to get back to normal.

Mike

On Sunday, 28 April 2019 12:26:02 UTC+1, mwall wrote:
>
> On Sunday, April 28, 2019 at 2:29:41 AM UTC-4, Redanman wrote:
>>
>> I had Debug=1 when it crashed.  I've gone back into the syslog archives 
>> and pulled out the section starting roughly 5 minutes before the last 
>> crash.  At that point everything seems to be running fine.  See attached.
>>
>
> i do not see the value 645138 anywhere in that log snippet.  could you see 
> if it is anywhere in the log prior to the exception you posted at the 
> beginning of this discussion?
>
> we're looking for it in one of the 'raw packet' lines
>
> also, you might want to query your database to see if there are any 
> strings in there.  if you find one, then the timestamp of the record with 
> the string should give you some idea of when in the logs to look
>
> m
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[mwall]

On Sunday, April 28, 2019 at 2:29:41 AM UTC-4, Redanman wrote:
>
> I had Debug=1 when it crashed.  I've gone back into the syslog archives 
> and pulled out the section starting roughly 5 minutes before the last 
> crash.  At that point everything seems to be running fine.  See attached.
>

i do not see the value 645138 anywhere in that log snippet.  could you see 
if it is anywhere in the log prior to the exception you posted at the 
beginning of this discussion?

we're looking for it in one of the 'raw packet' lines

also, you might want to query your database to see if there are any strings 
in there.  if you find one, then the timestamp of the record with the 
string should give you some idea of when in the logs to look

m

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[Redanman]

I had Debug=1 when it crashed.  I've gone back into the syslog archives and 
pulled out the section starting roughly 5 minutes before the last crash.  
At that point everything seems to be running fine.  See attached.
Mike

On Sunday, 28 April 2019 04:28:14 UTC+1, mwall wrote:
>
> could you set debug=1 then let weewx run until it crashes.  we need to see 
> the intercepted data and how those data are parsed and/or mapped to weewx 
> database fields.  tom is probably right - somehow the data are coming in as 
> a string instead of an integer.
>
> m
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Apr 25 18:55:11 raspberrypi weewx[331]: interceptor: MainThread: raw packet: 
{'wind_speed': 2.24, 'humidity_in': 48.0, 'temperature_in': 69.8, 'barometer': 
29.63, 'windchill': 57.4, 'dewpoint': 46.2, 'battery': 0.0, 'humidity_out': 
66.0, 'uv': 1.0, 'radiation': 298.29, 'rain': 0.0, 'dateTime': 1556214910, 
'temperature_out': 57.4, 'wind_dir': 165.0, 'rain_total': 4.16, 'usUnits': 1, 
'wind_gust': 4.92}
Apr 25 18:55:11 raspberrypi weewx[331]: interceptor: MainThread: mapped packet: 
{'barometer': 29.63, 'windchill': 57.4, 'dewpoint': 46.2, 'outHumidity': 66.0, 
'UV': 1.0, 'radiation': 298.29, 'rain': 0.0, 'dateTime': 1556214910, 'windDir': 
165.0, 'outTemp': 57.4, 'windSpeed': 2.24, 'txBatteryStatus': 0.0, 'inTemp': 
69.8, 'windGust': 4.92, 'inHumidity': 48.0, 'usUnits': 1}
Apr 25 18:55:21 raspberrypi weewx[331]: interceptor: MainThread: empty queue
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: ServerThread: GET: 
ID=Dummy&PASSWORD=&tempf=57.4&humidity=66&dewptf=46.2&windchillf=57.4&winddir=224&windspeedmph=4.70&windgustmph=7.61&rainin=0.00&dailyrainin=0.08&weeklyrainin=0.11&monthlyrainin=0.33&yearlyrainin=4.16&solarradiation=297.33&UV=1&indoortempf=69.8&indoorhumidity=48&baromin=29.64&lowbatt=0&dateutc=2019-04-25%2017:55:26&softwaretype=WH2600GEN_V2.2.7&action=updateraw&realtime=1&rtfreq=5
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: raw data: 
ID=Dummy&PASSWORD=dummy&tempf=57.4&humidity=66&dewptf=46.2&windchillf=57.4&winddir=224&windspeedmph=4.70&windgustmph=7.61&rainin=0.00&dailyrainin=0.08&weeklyrainin=0.11&monthlyrainin=0.33&yearlyrainin=4.16&solarradiation=297.33&UV=1&indoortempf=69.8&indoorhumidity=48&baromin=29.64&lowbatt=0&dateutc=2019-04-25%2017:55:26&softwaretype=WH2600GEN_V2.2.7&action=updateraw&realtime=1&rtfreq=5
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: using 
rain_total 4.16 from yearlyrainin
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: firmware 
WH2600GEN_V2.2.7: using baromin as barometer
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: ignored 
parameter rainin=0.00
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: ignored 
parameter monthlyrainin=0.33
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: ignored 
parameter realtime=1
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: ignored 
parameter rtfreq=5
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: ignored 
parameter PASSWORD=dummy
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: ignored 
parameter ID=Dummy
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: ignored 
parameter softwaretype=WH2600GEN_V2.2.7
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: ignored 
parameter action=updateraw
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: ignored 
parameter weeklyrainin=0.11
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: raw packet: 
{'wind_speed': 4.7, 'humidity_in': 48.0, 'temperature_in': 69.8, 'barometer': 
29.64, 'windchill': 57.4, 'dewpoint': 46.2, 'battery': 0.0, 'humidity_out': 
66.0, 'uv': 1.0, 'radiation': 297.33, 'rain': 0.0, 'dateTime': 1556214926, 
'temperature_out': 57.4, 'wind_dir': 224.0, 'rain_total': 4.16, 'usUnits': 1, 
'wind_gust': 7.61}
Apr 25 18:55:27 raspberrypi weewx[331]: interceptor: MainThread: mapped packet: 
{'barometer': 29.64, 'windchill': 57.4, 'dewpoint': 46.2, 'outHumidity': 66.0, 
'UV': 1.0, 'radiation': 297.33, 'rain': 0.0, 'dateTime': 1556214926, 'windDir': 
224.0, 'outTemp': 57.4, 'windSpeed': 4.7, 'txBatteryStatus': 0.0, 'inTemp': 
69.8, 'windGust': 7.61, 'inHumidity': 48.0, 'usUnits': 1}
Apr 25 18:55:27 raspberrypi weewx[331]: manager: Added record 2019-04-25 
18:55:28 BST (1556214928) to database 'cmon.sdb'
Apr 25 18:55:27 raspberrypi weewx[331]: manager: Added record 2019-04-25 
18:55:28 BST (1556214928) to daily summary in 'cmon.sdb'
Apr 25 18:55:37 raspberrypi weewx[331]: manager: Added record 2019-04-25 
18:55:00 BST (1556214900) to database 'weewx.sdb'
Apr 25 18:55:37 raspberrypi weewx[331]: manager: Added record 2019-04-25 
18:55:00 

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[mwall]

could you set debug=1 then let weewx run until it crashes.  we need to see 
the intercepted data and how those data are parsed and/or mapped to weewx 
database fields.  tom is probably right - somehow the data are coming in as 
a string instead of an integer.

m

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[Redanman]

For me it's an Aercus WeatherSleuth.  The section of my config pertaining 
to this device is :

[Interceptor]
# This section is for the network traffic interceptor driver.

# The driver to use:
driver = user.interceptor
port = 55
device_type = observer


On Saturday, 27 April 2019 01:14:15 UTC+1, mwall wrote:
>
> On Friday, April 26, 2019 at 6:34:50 PM UTC-4, pliggen.is wrote:
>>
>> I also got the issue two days ago. Strange because I've been running for 
>> quite some time now and just the other day was my first crash. 
>>
>
> what kind of weather station?
>
> what is your interceptor configuration?  (please post the [Interceptor] 
> section from your weewx configuration file)
>
> m
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[mwall]

On Friday, April 26, 2019 at 6:34:50 PM UTC-4, pliggen.is wrote:
>
> I also got the issue two days ago. Strange because I've been running for 
> quite some time now and just the other day was my first crash. 
>

what kind of weather station?

what is your interceptor configuration?  (please post the [Interceptor] 
section from your weewx configuration file)

m

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing with error associated with ScalarStats.addHiLo

[pliggen . is]

I also got the issue two days ago. Strange because I've been running for 
quite some time now and just the other day was my first crash. 

Here's my part of the log when it happened:

Apr 25 07:31:07 raspberrypi weewx[25759]: engine: Main loop exiting. 
Shutting engine down.
Apr 25 07:31:07 raspberrypi weewx[25759]: engine: Shutting down StdReport 
thread
Apr 25 07:31:07 raspberrypi weewx[25759]: engine: Caught unrecoverable 
exception in engine:
Apr 25 07:31:07 raspberrypi weewx[25759]:   accum: 
ScalarStats.addHiLo expected float or int, got 2190515
Apr 25 07:31:07 raspberrypi weewx[25759]:   Traceback (most recent 
call last):
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/weewx/engine.py", line 890, in main
Apr 25 07:31:07 raspberrypi weewx[25759]:   engine.run()
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/weewx/engine.py", line 202, in run
Apr 25 07:31:07 raspberrypi weewx[25759]:   
self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
Apr 25 07:31:07 raspberrypi weewx[25759]:   callback(event)
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/weewx/engine.py", line 574, in post_loop
Apr 25 07:31:07 raspberrypi weewx[25759]:   
self._software_catchup()
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/weewx/engine.py", line 646, in _software_catchup
Apr 25 07:31:07 raspberrypi weewx[25759]:   
self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD, 
record=record, origin='software'))
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent
Apr 25 07:31:07 raspberrypi weewx[25759]:   callback(event)
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/user/cmon.py", line 706, in new_archive_record
Apr 25 07:31:07 raspberrypi weewx[25759]:   
self.save_data(self.get_data(now, self.last_ts))
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/user/cmon.py", line 713, in save_data
Apr 25 07:31:07 raspberrypi weewx[25759]:   
self.dbm.addRecord(record)
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/weewx/manager.py", line 246, in addRecord
Apr 25 07:31:07 raspberrypi weewx[25759]:   
self._addSingleRecord(record, cursor, log_level)
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/weewx/manager.py", line 1216, in _addSingleRecord
Apr 25 07:31:07 raspberrypi weewx[25759]:   
_day_summary.addRecord(record, weight=_weight)
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/weewx/accum.py", line 256, in addRecord
Apr 25 07:31:07 raspberrypi weewx[25759]:   func(self, record, 
obs_type, add_hilo, weight)
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/weewx/accum.py", line 314, in add_value
Apr 25 07:31:07 raspberrypi weewx[25759]:   
self[obs_type].addHiLo(val, record['dateTime'])
Apr 25 07:31:07 raspberrypi weewx[25759]: File 
"/usr/share/weewx/weewx/accum.py", line 77, in addHiLo
Apr 25 07:31:07 raspberrypi weewx[25759]:   raise 
ValueError("accum: ScalarStats.addHiLo expected float or int, got %s" % val)
Apr 25 07:31:07 raspberrypi weewx[25759]:   ValueError: accum: 
ScalarStats.addHiLo expected float or int, got 2190515
Apr 25 07:31:07 raspberrypi weewx[25759]:   Exiting.

/Micael

Den fredag 26 april 2019 kl. 12:57:48 UTC+2 skrev Redanman:
>
> Thank you for any possible insights into what might be causing Weewx to 
> occasionally crash.  The relevant part of the syslog is :
>
> Apr 26 08:20:32 raspberrypi weewx[7715]: File 
> "/usr/share/weewx/weewx/manager.py", line 1216, in _addSingleRecord
> Apr 26 08:20:32 raspberrypi weewx[7715]:   
> _day_summary.addRecord(record, weight=_weight)
> Apr 26 08:20:32 raspberrypi weewx[7715]: File 
> "/usr/share/weewx/weewx/accum.py", line 256, in addRecord
> Apr 26 08:20:32 raspberrypi weewx[7715]:   func(self, record, 
> obs_type, add_hilo, weight)
> Apr 26 08:20:32 raspberrypi weewx[7715]: File 
> "/usr/share/weewx/weewx/accum.py", line 314, in add_value
> Apr 26 08:20:32 raspberrypi weewx[7715]:   
> self[obs_type].addHiLo(val, record['dateTime'])
> Apr 26 08:20:32 raspberrypi weewx[7715]: File 
> "/usr/share/weewx/weewx/accum.py", line 77, in addHiLo
> Apr 26 08:20:32 raspberrypi weewx[7715]:   raise 
> ValueError("accum: ScalarStats.addHiLo expected float or int, got %s" % val)
> Apr 26 08:20:32 raspberrypi weewx[7715]:   ValueError: accum: 
> ScalarStats.addHiLo expected float or int, got 6451

Re: [weewx-user] Re: Weewx crashing

[Thomas Carlin]

That should do the trick.  Thanks Tom!

Enjoy Thanksgiving tomorrow all!


On Tuesday, November 21, 2017 at 5:29:12 AM UTC-7, Tom Keffer wrote:
>
> Good point. I went with
>
> val != val
>
>
> Commit  0bce71e 
> 
>  
>
> -tk
>
> On Mon, Nov 20, 2017 at 9:47 PM, Thomas Carlin  > wrote:
>
>> True, didn't think about that.  It's dirty, but 
>>
>> str(val) == 'nan'
>>
>> would work also.  Aside from isnan(), that is the most readable solution 
>> that I am aware of.
>>
>>
>> On Monday, November 20, 2017 at 8:33:28 PM UTC-7, gjr80 wrote:
>>>
>>> Might have to be careful we don't break python 2.5 compatibility, I 
>>> think isnan() was introduced in python 2.6.
>>>
>>> Gary
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Weewx crashing

[Thomas Keffer]

Good point. I went with

val != val


Commit  0bce71e



-tk

On Mon, Nov 20, 2017 at 9:47 PM, Thomas Carlin 
wrote:

> True, didn't think about that.  It's dirty, but
>
> str(val) == 'nan'
>
> would work also.  Aside from isnan(), that is the most readable solution
> that I am aware of.
>
>
> On Monday, November 20, 2017 at 8:33:28 PM UTC-7, gjr80 wrote:
>>
>> Might have to be careful we don't break python 2.5 compatibility, I think
>> isnan() was introduced in python 2.6.
>>
>> Gary
>>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Weewx crashing

[Thomas Carlin]

True, didn't think about that.  It's dirty, but 

str(val) == 'nan'

would work also.  Aside from isnan(), that is the most readable solution 
that I am aware of.

On Monday, November 20, 2017 at 8:33:28 PM UTC-7, gjr80 wrote:
>
> Might have to be careful we don't break python 2.5 compatibility, I think 
> isnan() was introduced in python 2.6.
>
> Gary
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Weewx crashing

[gjr80]

Might have to be careful we don't break python 2.5 compatibility, I think 
isnan() was introduced in python 2.6.

Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Weewx crashing

[Thomas Carlin]

I don't think this will catch float('nan'), which is how I was bit this 
time.  

val = float('nan')
>>> isinstance(val, (float, int))
True
>>>
>>> type(val)



May I recommend as an alternative, slightly more robust solution:

def addSum(self, val, weight=1):
 """Add a scalar value to my running sum and count."""
 if val is not None:
+if (not isinstance(val, (float, int))) or (math.isnan(val)):
+raise ValueError("accum: Expected float or int, got %s" % 
val)
 self.sum += val
 self.count   += 1
 self.wsum+= val * weight


On Monday, November 20, 2017 at 4:16:56 PM UTC-7, Tom Keffer wrote:
>
> Implemented at commit 40d7a2c 
> 
> .
>
> -tk
>
> On Mon, Nov 20, 2017 at 4:45 PM, gjr80 > 
> wrote:
>
>> Well I am embarrased to say that to say I did not know about nan/NaN and 
>> python, today's lesson I guess.
>>
>> Gary
>>
>> On Tuesday, 21 November 2017 01:30:57 UTC+10, Thomas Carlin wrote:
>>>
>>> Tom,
>>> Your interpretation is exactly correct, and yes it is a coding error on 
>>> my side.  My thought was just a single sentence there about the Weewx 
>>> engine not doing any sanitation, and that being the responsibility of the 
>>> driver.  If not, no worries, just a thought.
>>>
>>> Gary,
>>> No worries on the delay.  You got me going in the right direction, and 
>>> we got it figured out! It's not like you get paid for doing this.  Until 
>>> yesterday, the extent of the sanitation that I was doing on the incoming 
>>> data was running float(data), and catching any ValueError exception.  Now, 
>>> because nan is a special 'string', when you run float('nan'), it returns 
>>> NaN, but now it is the Python 'number' NaN.  When you add a number to NaN, 
>>> your result is always NaN.  When this is stuffed into the database, 
>>> apparently it is interpreted as SQL NULL.  Next time the loop runs, Python 
>>> queries the database, and interprets SQL NULL as None, and then we get the 
>>> type mismatch.  
>>>
>>> I have added some more sanitation for this, and we are back up and 
>>> running, and it *should* conform to expected behavior.  
>>>
>>> Does that make sense?
>>>
>>> Thank you for the input on the driver.  I have a laundry list of things 
>>> that I would like to add to make it more robust, and I'll add the loop 
>>> output to that.  I may be converting to a queue as well, simply due to the 
>>> number of sensors that are being polled, and the "couple of seconds" note 
>>> in the Second Datasource section of the docs.  But that is a project for 
>>> another day!
>>>
>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Weewx crashing

[Thomas Keffer]

Implemented at commit 40d7a2c

.

-tk

On Mon, Nov 20, 2017 at 4:45 PM, gjr80  wrote:

> Well I am embarrased to say that to say I did not know about nan/NaN and
> python, today's lesson I guess.
>
> Gary
>
> On Tuesday, 21 November 2017 01:30:57 UTC+10, Thomas Carlin wrote:
>>
>> Tom,
>> Your interpretation is exactly correct, and yes it is a coding error on
>> my side.  My thought was just a single sentence there about the Weewx
>> engine not doing any sanitation, and that being the responsibility of the
>> driver.  If not, no worries, just a thought.
>>
>> Gary,
>> No worries on the delay.  You got me going in the right direction, and we
>> got it figured out! It's not like you get paid for doing this.  Until
>> yesterday, the extent of the sanitation that I was doing on the incoming
>> data was running float(data), and catching any ValueError exception.  Now,
>> because nan is a special 'string', when you run float('nan'), it returns
>> NaN, but now it is the Python 'number' NaN.  When you add a number to NaN,
>> your result is always NaN.  When this is stuffed into the database,
>> apparently it is interpreted as SQL NULL.  Next time the loop runs, Python
>> queries the database, and interprets SQL NULL as None, and then we get the
>> type mismatch.
>>
>> I have added some more sanitation for this, and we are back up and
>> running, and it *should* conform to expected behavior.
>>
>> Does that make sense?
>>
>> Thank you for the input on the driver.  I have a laundry list of things
>> that I would like to add to make it more robust, and I'll add the loop
>> output to that.  I may be converting to a queue as well, simply due to the
>> number of sensors that are being polled, and the "couple of seconds" note
>> in the Second Datasource section of the docs.  But that is a project for
>> another day!
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Weewx crashing

[gjr80]

Well I am embarrased to say that to say I did not know about nan/NaN and 
python, today's lesson I guess.

Gary

On Tuesday, 21 November 2017 01:30:57 UTC+10, Thomas Carlin wrote:
>
> Tom,
> Your interpretation is exactly correct, and yes it is a coding error on my 
> side.  My thought was just a single sentence there about the Weewx engine 
> not doing any sanitation, and that being the responsibility of the driver.  
> If not, no worries, just a thought.
>
> Gary,
> No worries on the delay.  You got me going in the right direction, and we 
> got it figured out! It's not like you get paid for doing this.  Until 
> yesterday, the extent of the sanitation that I was doing on the incoming 
> data was running float(data), and catching any ValueError exception.  Now, 
> because nan is a special 'string', when you run float('nan'), it returns 
> NaN, but now it is the Python 'number' NaN.  When you add a number to NaN, 
> your result is always NaN.  When this is stuffed into the database, 
> apparently it is interpreted as SQL NULL.  Next time the loop runs, Python 
> queries the database, and interprets SQL NULL as None, and then we get the 
> type mismatch.  
>
> I have added some more sanitation for this, and we are back up and 
> running, and it *should* conform to expected behavior.  
>
> Does that make sense?
>
> Thank you for the input on the driver.  I have a laundry list of things 
> that I would like to add to make it more robust, and I'll add the loop 
> output to that.  I may be converting to a queue as well, simply due to the 
> number of sensors that are being polled, and the "couple of seconds" note 
> in the Second Datasource section of the docs.  But that is a project for 
> another day!
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Weewx crashing

[Thomas Keffer]

Thanks for the explanation. I think I understand the failure mode now.

One thing we could do is put a check in ScalarStats.addSum() to make sure
the incoming value is, in fact, a number (and not a String, None, or NaN).
I don't think it's a good idea to just swallow and suppress the error, but
this way, it would be immediately obvious who and what the culprit is. The
patch would look something like this:

diff --git a/bin/weewx/accum.py b/bin/weewx/accum.py
index 26bf243..ee54db3 100644
--- a/bin/weewx/accum.py
+++ b/bin/weewx/accum.py
@@ -85,6 +85,8 @@
 def addSum(self, val, weight=1):
 """Add a scalar value to my running sum and count."""
 if val is not None:
+if not isinstance(val, (float, int)):
+raise ValueError("accum: Expected float or int, got %s" %
val)
 self.sum += val
 self.count   += 1
 self.wsum+= val * weight

-tk


On Mon, Nov 20, 2017 at 9:30 AM, Thomas Carlin 
wrote:

> Tom,
> Your interpretation is exactly correct, and yes it is a coding error on my
> side.  My thought was just a single sentence there about the Weewx engine
> not doing any sanitation, and that being the responsibility of the driver.
> If not, no worries, just a thought.
>
> Gary,
> No worries on the delay.  You got me going in the right direction, and we
> got it figured out! It's not like you get paid for doing this.  Until
> yesterday, the extent of the sanitation that I was doing on the incoming
> data was running float(data), and catching any ValueError exception.  Now,
> because nan is a special 'string', when you run float('nan'), it returns
> NaN, but now it is the Python 'number' NaN.  When you add a number to NaN,
> your result is always NaN.  When this is stuffed into the database,
> apparently it is interpreted as SQL NULL.  Next time the loop runs, Python
> queries the database, and interprets SQL NULL as None, and then we get the
> type mismatch.
>
> I have added some more sanitation for this, and we are back up and
> running, and it *should* conform to expected behavior.
>
> Does that make sense?
>
> Thank you for the input on the driver.  I have a laundry list of things
> that I would like to add to make it more robust, and I'll add the loop
> output to that.  I may be converting to a queue as well, simply due to the
> number of sensors that are being polled, and the "couple of seconds" note
> in the Second Datasource section of the docs.  But that is a project for
> another day!
>
> On Mon, Nov 20, 2017 at 12:53 AM, gjr80  wrote:
>
>> Thomas C,
>>
>> Sorry I did not respond to your earlier posts after mine, the extract you
>> provided does indeed show a daily summary record with sum and wsum
>> fields as None (well the db fields contain nothing but they will end up
>> as python None values within weeWX). At the moment I have still not been
>> able to fathom the circumstances that would cause this.
>>
>> One thing about your driver though. If you are using the default weeWX
>> schema, or a modified version of the default schema to which you have only
>> added numeric (read SQL type REAL) fields, then your driver must only emit
>> numeric values or the python None value. If you emit strings it will
>> cause an exception to be thrown within weeWX. The default weeWX behaviour
>> is if the sensor is present and emitting values then the driver should emit
>> those values as numeric values in the approriate loop packet/archive record
>> field, if the sensor is present but offline then the driver emits the
>> python None value in the field concerned and if the sensor does not
>> exist then the driver should not emit anything for that sensor ie the field
>> is excluded from the loop packet/archive record.
>>
>> Note. Technically there is no reason that you cannot add a string to a
>> loop packet or an archive record but if it is placed in a field that weeWX
>> tries to 'accumulate' it will cause an exception.
>>
>> I still don't see how your driver can cause the exception you first
>> posted (emitting a string will cause a similar but clearly different
>> exception), but it may be worthwhile reworking your driver to conform with
>> the weeWX expected behaviour. It might also be worthwhile instrumenting
>> your driver to drop the emitted loop packet to the log so we can see
>> exactly what is coming out of the driver in the leadup to the exception.
>>
>> Gary
>>
>>
>> On Monday, 20 November 2017 08:33:28 UTC+10, Thomas Carlin wrote:
>>>
>>> Hey Tom
>>>
>>> At this point in the code 'nan' is not equivalent to 'Not a Number', it
>>> is simply a string that is pulled out of an http page, not generated by
>>> python.  Originally, it is generated on my Arduino when data cannot be
>>> pulled from the temperature sensor.  Unfortunately in this case when you
>>> run float("nan") in python, it returns nan, and that was the extent of my
>>> sanitization.  I'll write a condition into my driver to catch that, and
>>> should be good to g

Re: [weewx-user] Re: Weewx crashing

[Thomas Carlin]

Tom,
Your interpretation is exactly correct, and yes it is a coding error on my
side.  My thought was just a single sentence there about the Weewx engine
not doing any sanitation, and that being the responsibility of the driver.
If not, no worries, just a thought.

Gary,
No worries on the delay.  You got me going in the right direction, and we
got it figured out! It's not like you get paid for doing this.  Until
yesterday, the extent of the sanitation that I was doing on the incoming
data was running float(data), and catching any ValueError exception.  Now,
because nan is a special 'string', when you run float('nan'), it returns
NaN, but now it is the Python 'number' NaN.  When you add a number to NaN,
your result is always NaN.  When this is stuffed into the database,
apparently it is interpreted as SQL NULL.  Next time the loop runs, Python
queries the database, and interprets SQL NULL as None, and then we get the
type mismatch.

I have added some more sanitation for this, and we are back up and running,
and it *should* conform to expected behavior.

Does that make sense?

Thank you for the input on the driver.  I have a laundry list of things
that I would like to add to make it more robust, and I'll add the loop
output to that.  I may be converting to a queue as well, simply due to the
number of sensors that are being polled, and the "couple of seconds" note
in the Second Datasource section of the docs.  But that is a project for
another day!

On Mon, Nov 20, 2017 at 12:53 AM, gjr80  wrote:

> Thomas C,
>
> Sorry I did not respond to your earlier posts after mine, the extract you
> provided does indeed show a daily summary record with sum and wsum fields
> as None (well the db fields contain nothing but they will end up as
> python None values within weeWX). At the moment I have still not been
> able to fathom the circumstances that would cause this.
>
> One thing about your driver though. If you are using the default weeWX
> schema, or a modified version of the default schema to which you have only
> added numeric (read SQL type REAL) fields, then your driver must only emit
> numeric values or the python None value. If you emit strings it will
> cause an exception to be thrown within weeWX. The default weeWX behaviour
> is if the sensor is present and emitting values then the driver should emit
> those values as numeric values in the approriate loop packet/archive record
> field, if the sensor is present but offline then the driver emits the
> python None value in the field concerned and if the sensor does not exist
> then the driver should not emit anything for that sensor ie the field is
> excluded from the loop packet/archive record.
>
> Note. Technically there is no reason that you cannot add a string to a
> loop packet or an archive record but if it is placed in a field that weeWX
> tries to 'accumulate' it will cause an exception.
>
> I still don't see how your driver can cause the exception you first posted
> (emitting a string will cause a similar but clearly different exception),
> but it may be worthwhile reworking your driver to conform with the weeWX
> expected behaviour. It might also be worthwhile instrumenting your driver
> to drop the emitted loop packet to the log so we can see exactly what is
> coming out of the driver in the leadup to the exception.
>
> Gary
>
>
> On Monday, 20 November 2017 08:33:28 UTC+10, Thomas Carlin wrote:
>>
>> Hey Tom
>>
>> At this point in the code 'nan' is not equivalent to 'Not a Number', it
>> is simply a string that is pulled out of an http page, not generated by
>> python.  Originally, it is generated on my Arduino when data cannot be
>> pulled from the temperature sensor.  Unfortunately in this case when you
>> run float("nan") in python, it returns nan, and that was the extent of my
>> sanitization.  I'll write a condition into my driver to catch that, and
>> should be good to go.
>>
>> On Sunday, November 19, 2017 at 2:14:59 PM UTC-7, Tom Keffer wrote:
>>>
>>> But, how did you create a NaN? They are not returned from any of the
>>> Python mathematical functions.
>>>
>>> -tk
>>>
>>> On Sun, Nov 19, 2017 at 2:04 PM, Thomas Carlin 
>>> wrote:
>>>
 I'm pretty sure this is what is happening.  Take a look at the output
 below:

 ...
 DB Entry: extraTemp3
 Value: nan
 DB Entry: extraHumid3
 Value: nan
 ...
 Unable to access http://192.168.2.105
 REC:2017-11-19 12:38:00 MST (1511120280) ..., *extraHumid3: nan*,
 extraTemp1: 55.58, *extraTemp3: nan*, heatindex: 31.7143461695...

 The next attempt to add data to the database resulted in the program
 crashing due to a type mismatch.

 May I request that a blurb be added to the documentation here:
 http://www.weewx.com/docs/customizing.htm#Adding_2nd_source about
 sanitizing the data before it is fed to event.record()?


 On Sunday, November 19, 2017 at 10:59:29 AM UTC-7, Thomas Carlin wrote:
>
> The plot thickens

Re: [weewx-user] Re: Weewx crashing

[gjr80]

Thomas C,

Sorry I did not respond to your earlier posts after mine, the extract you 
provided does indeed show a daily summary record with sum and wsum fields 
as None (well the db fields contain nothing but they will end up as python 
None values within weeWX). At the moment I have still not been able to 
fathom the circumstances that would cause this.

One thing about your driver though. If you are using the default weeWX 
schema, or a modified version of the default schema to which you have only 
added numeric (read SQL type REAL) fields, then your driver must only emit 
numeric values or the python None value. If you emit strings it will cause 
an exception to be thrown within weeWX. The default weeWX behaviour is if 
the sensor is present and emitting values then the driver should emit those 
values as numeric values in the approriate loop packet/archive record 
field, if the sensor is present but offline then the driver emits the 
python None value in the field concerned and if the sensor does not exist 
then the driver should not emit anything for that sensor ie the field is 
excluded from the loop packet/archive record.

Note. Technically there is no reason that you cannot add a string to a loop 
packet or an archive record but if it is placed in a field that weeWX tries 
to 'accumulate' it will cause an exception.

I still don't see how your driver can cause the exception you first posted 
(emitting a string will cause a similar but clearly different exception), 
but it may be worthwhile reworking your driver to conform with the weeWX 
expected behaviour. It might also be worthwhile instrumenting your driver 
to drop the emitted loop packet to the log so we can see exactly what is 
coming out of the driver in the leadup to the exception.

Gary

On Monday, 20 November 2017 08:33:28 UTC+10, Thomas Carlin wrote:
>
> Hey Tom
>
> At this point in the code 'nan' is not equivalent to 'Not a Number', it is 
> simply a string that is pulled out of an http page, not generated by 
> python.  Originally, it is generated on my Arduino when data cannot be 
> pulled from the temperature sensor.  Unfortunately in this case when you 
> run float("nan") in python, it returns nan, and that was the extent of my 
> sanitization.  I'll write a condition into my driver to catch that, and 
> should be good to go.
>
> On Sunday, November 19, 2017 at 2:14:59 PM UTC-7, Tom Keffer wrote:
>>
>> But, how did you create a NaN? They are not returned from any of the 
>> Python mathematical functions.
>>
>> -tk
>>
>> On Sun, Nov 19, 2017 at 2:04 PM, Thomas Carlin  
>> wrote:
>>
>>> I'm pretty sure this is what is happening.  Take a look at the output 
>>> below:
>>>
>>> ...
>>> DB Entry: extraTemp3
>>> Value: nan
>>> DB Entry: extraHumid3
>>> Value: nan
>>> ...
>>> Unable to access http://192.168.2.105
>>> REC:2017-11-19 12:38:00 MST (1511120280) ..., *extraHumid3: nan*, 
>>> extraTemp1: 55.58, *extraTemp3: nan*, heatindex: 31.7143461695...
>>>
>>> The next attempt to add data to the database resulted in the program 
>>> crashing due to a type mismatch.  
>>>
>>> May I request that a blurb be added to the documentation here: 
>>> http://www.weewx.com/docs/customizing.htm#Adding_2nd_source about 
>>> sanitizing the data before it is fed to event.record()?
>>>
>>>
>>> On Sunday, November 19, 2017 at 10:59:29 AM UTC-7, Thomas Carlin wrote:

 The plot thickens.  Right before I went to bed this morning, I checked 
 the current daily summary record for extraTemp3, and it looked as you 
 would 
 expect it to.  It had data in it for both sum and wsum, unlike the one 
 posted above.  Roughly an hour later, Weewx crashed with the same error, 
 and this morning when I went to look into it, the current record, same one 
 as last night, no longer had data in it for the same fields.  

 After looking a little closer, I noticed that the extraHumid3, which I 
 created, did exactly the same thing as extraTemp3.  Since both these 
 pieces 
 of data come from the same DHT11 sensor, my current theory is that under 
 the wrong conditions, my sensor is spitting out garbage that is not being 
 properly sanitized, which somehow empties those fields.  Then next time 
 the 
 loop runs, everything falls apart.  I'm setting up a sandbox right now so 
 I 
 can play with it without screwing with my 'production' system.  Just 
 hoping 
 the sandbox crashes like the current system does!

>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to weewx-user+...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
t

Re: [weewx-user] Re: Weewx crashing

[Thomas Keffer]

So, if I'm understanding you correctly, the Arduino produces numbers as
strings. It can also produce the string "nan" when no data are available.
So, when it came time to convert the string to a floating point number in
your driver, you effectively had float("nan"), which is how the Python NaN
was produced. Is that it?

If that's the case, that just sounds like a run-of-the-mill programming
error. There's no end of things like that which could go wrong. We can't
list them all in the Customizing Guide.

-tk


On Sun, Nov 19, 2017 at 4:33 PM, Thomas Carlin 
wrote:

> Hey Tom
>
> At this point in the code 'nan' is not equivalent to 'Not a Number', it is
> simply a string that is pulled out of an http page, not generated by
> python.  Originally, it is generated on my Arduino when data cannot be
> pulled from the temperature sensor.  Unfortunately in this case when you
> run float("nan") in python, it returns nan, and that was the extent of my
> sanitization.  I'll write a condition into my driver to catch that, and
> should be good to go.
>
> On Sunday, November 19, 2017 at 2:14:59 PM UTC-7, Tom Keffer wrote:
>>
>> But, how did you create a NaN? They are not returned from any of the
>> Python mathematical functions.
>>
>> -tk
>>
>> On Sun, Nov 19, 2017 at 2:04 PM, Thomas Carlin 
>> wrote:
>>
>>> I'm pretty sure this is what is happening.  Take a look at the output
>>> below:
>>>
>>> ...
>>> DB Entry: extraTemp3
>>> Value: nan
>>> DB Entry: extraHumid3
>>> Value: nan
>>> ...
>>> Unable to access http://192.168.2.105
>>> REC:2017-11-19 12:38:00 MST (1511120280) ..., *extraHumid3: nan*,
>>> extraTemp1: 55.58, *extraTemp3: nan*, heatindex: 31.7143461695...
>>>
>>> The next attempt to add data to the database resulted in the program
>>> crashing due to a type mismatch.
>>>
>>> May I request that a blurb be added to the documentation here:
>>> http://www.weewx.com/docs/customizing.htm#Adding_2nd_source about
>>> sanitizing the data before it is fed to event.record()?
>>>
>>>
>>> On Sunday, November 19, 2017 at 10:59:29 AM UTC-7, Thomas Carlin wrote:

 The plot thickens.  Right before I went to bed this morning, I checked
 the current daily summary record for extraTemp3, and it looked as you would
 expect it to.  It had data in it for both sum and wsum, unlike the one
 posted above.  Roughly an hour later, Weewx crashed with the same error,
 and this morning when I went to look into it, the current record, same one
 as last night, no longer had data in it for the same fields.

 After looking a little closer, I noticed that the extraHumid3, which I
 created, did exactly the same thing as extraTemp3.  Since both these pieces
 of data come from the same DHT11 sensor, my current theory is that under
 the wrong conditions, my sensor is spitting out garbage that is not being
 properly sanitized, which somehow empties those fields.  Then next time the
 loop runs, everything falls apart.  I'm setting up a sandbox right now so I
 can play with it without screwing with my 'production' system.  Just hoping
 the sandbox crashes like the current system does!

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-user+...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Weewx crashing

[Thomas Carlin]

Hey Tom

At this point in the code 'nan' is not equivalent to 'Not a Number', it is 
simply a string that is pulled out of an http page, not generated by 
python.  Originally, it is generated on my Arduino when data cannot be 
pulled from the temperature sensor.  Unfortunately in this case when you 
run float("nan") in python, it returns nan, and that was the extent of my 
sanitization.  I'll write a condition into my driver to catch that, and 
should be good to go.

On Sunday, November 19, 2017 at 2:14:59 PM UTC-7, Tom Keffer wrote:
>
> But, how did you create a NaN? They are not returned from any of the 
> Python mathematical functions.
>
> -tk
>
> On Sun, Nov 19, 2017 at 2:04 PM, Thomas Carlin  > wrote:
>
>> I'm pretty sure this is what is happening.  Take a look at the output 
>> below:
>>
>> ...
>> DB Entry: extraTemp3
>> Value: nan
>> DB Entry: extraHumid3
>> Value: nan
>> ...
>> Unable to access http://192.168.2.105
>> REC:2017-11-19 12:38:00 MST (1511120280) ..., *extraHumid3: nan*, 
>> extraTemp1: 55.58, *extraTemp3: nan*, heatindex: 31.7143461695...
>>
>> The next attempt to add data to the database resulted in the program 
>> crashing due to a type mismatch.  
>>
>> May I request that a blurb be added to the documentation here: 
>> http://www.weewx.com/docs/customizing.htm#Adding_2nd_source about 
>> sanitizing the data before it is fed to event.record()?
>>
>>
>> On Sunday, November 19, 2017 at 10:59:29 AM UTC-7, Thomas Carlin wrote:
>>>
>>> The plot thickens.  Right before I went to bed this morning, I checked 
>>> the current daily summary record for extraTemp3, and it looked as you would 
>>> expect it to.  It had data in it for both sum and wsum, unlike the one 
>>> posted above.  Roughly an hour later, Weewx crashed with the same error, 
>>> and this morning when I went to look into it, the current record, same one 
>>> as last night, no longer had data in it for the same fields.  
>>>
>>> After looking a little closer, I noticed that the extraHumid3, which I 
>>> created, did exactly the same thing as extraTemp3.  Since both these pieces 
>>> of data come from the same DHT11 sensor, my current theory is that under 
>>> the wrong conditions, my sensor is spitting out garbage that is not being 
>>> properly sanitized, which somehow empties those fields.  Then next time the 
>>> loop runs, everything falls apart.  I'm setting up a sandbox right now so I 
>>> can play with it without screwing with my 'production' system.  Just hoping 
>>> the sandbox crashes like the current system does!
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Weewx crashing

[Thomas Keffer]

But, how did you create a NaN? They are not returned from any of the Python
mathematical functions.

-tk

On Sun, Nov 19, 2017 at 2:04 PM, Thomas Carlin 
wrote:

> I'm pretty sure this is what is happening.  Take a look at the output
> below:
>
> ...
> DB Entry: extraTemp3
> Value: nan
> DB Entry: extraHumid3
> Value: nan
> ...
> Unable to access http://192.168.2.105
> REC:2017-11-19 12:38:00 MST (1511120280) ..., *extraHumid3: nan*,
> extraTemp1: 55.58, *extraTemp3: nan*, heatindex: 31.7143461695
> <(714)%20346-1695>...
>
> The next attempt to add data to the database resulted in the program
> crashing due to a type mismatch.
>
> May I request that a blurb be added to the documentation here:
> http://www.weewx.com/docs/customizing.htm#Adding_2nd_source about
> sanitizing the data before it is fed to event.record()?
>
>
> On Sunday, November 19, 2017 at 10:59:29 AM UTC-7, Thomas Carlin wrote:
>>
>> The plot thickens.  Right before I went to bed this morning, I checked
>> the current daily summary record for extraTemp3, and it looked as you would
>> expect it to.  It had data in it for both sum and wsum, unlike the one
>> posted above.  Roughly an hour later, Weewx crashed with the same error,
>> and this morning when I went to look into it, the current record, same one
>> as last night, no longer had data in it for the same fields.
>>
>> After looking a little closer, I noticed that the extraHumid3, which I
>> created, did exactly the same thing as extraTemp3.  Since both these pieces
>> of data come from the same DHT11 sensor, my current theory is that under
>> the wrong conditions, my sensor is spitting out garbage that is not being
>> properly sanitized, which somehow empties those fields.  Then next time the
>> loop runs, everything falls apart.  I'm setting up a sandbox right now so I
>> can play with it without screwing with my 'production' system.  Just hoping
>> the sandbox crashes like the current system does!
>>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing

[Thomas Carlin]

I'm pretty sure this is what is happening.  Take a look at the output below:

...
DB Entry: extraTemp3
Value: nan
DB Entry: extraHumid3
Value: nan
...
Unable to access http://192.168.2.105
REC:2017-11-19 12:38:00 MST (1511120280) ..., *extraHumid3: nan*, 
extraTemp1: 55.58, *extraTemp3: nan*, heatindex: 31.7143461695...

The next attempt to add data to the database resulted in the program 
crashing due to a type mismatch.  

May I request that a blurb be added to the documentation here: 
http://www.weewx.com/docs/customizing.htm#Adding_2nd_source about 
sanitizing the data before it is fed to event.record()?

On Sunday, November 19, 2017 at 10:59:29 AM UTC-7, Thomas Carlin wrote:
>
> The plot thickens.  Right before I went to bed this morning, I checked the 
> current daily summary record for extraTemp3, and it looked as you would 
> expect it to.  It had data in it for both sum and wsum, unlike the one 
> posted above.  Roughly an hour later, Weewx crashed with the same error, 
> and this morning when I went to look into it, the current record, same one 
> as last night, no longer had data in it for the same fields.  
>
> After looking a little closer, I noticed that the extraHumid3, which I 
> created, did exactly the same thing as extraTemp3.  Since both these pieces 
> of data come from the same DHT11 sensor, my current theory is that under 
> the wrong conditions, my sensor is spitting out garbage that is not being 
> properly sanitized, which somehow empties those fields.  Then next time the 
> loop runs, everything falls apart.  I'm setting up a sandbox right now so I 
> can play with it without screwing with my 'production' system.  Just hoping 
> the sandbox crashes like the current system does!
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing

[Thomas Carlin]

The plot thickens.  Right before I went to bed this morning, I checked the 
current daily summary record for extraTemp3, and it looked as you would 
expect it to.  It had data in it for both sum and wsum, unlike the one 
posted above.  Roughly an hour later, Weewx crashed with the same error, 
and this morning when I went to look into it, the current record, same one 
as last night, no longer had data in it for the same fields.  

After looking a little closer, I noticed that the extraHumid3, which I 
created, did exactly the same thing as extraTemp3.  Since both these pieces 
of data come from the same DHT11 sensor, my current theory is that under 
the wrong conditions, my sensor is spitting out garbage that is not being 
properly sanitized, which somehow empties those fields.  Then next time the 
loop runs, everything falls apart.  I'm setting up a sandbox right now so I 
can play with it without screwing with my 'production' system.  Just hoping 
the sandbox crashes like the current system does!

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing

[Thomas Carlin]

Well, so much for that theory.   About 7:35p it crashed again.  But 
unplugging the device that generates that data has allowed it to run 
normally again.  However, based on your response the last record in this 
output would be the culprit then correct?  If my count is correct, both sum 
and wsum would be considered None when the accumulator is initialized.  Any 
idea what could cause this behavior?


sqlite> select * from archive_day_extraTemp3 where datetime >= 151064;
dateTime|min|mintime|max|maxtime|sum|count|wsum|sumtime
1510642800|70.52|1510728900|78.26|1510721100|2935.64|40|880692.0|12000
1510729200|65.48|1510746000|78.08|1510779600|16512.56|232|4953768.0|69600
1510815600|67.28|1510842900|71.06|1510842000|2934.08|43|880224.0|12900
1510902000|0.0|0|0.0|0
1510988400|74.48|1511058300|83.12|1511039400||45||13500
sqlite>




sqlite> select dateTime, inTemp, extraTemp3 from archive where datetime >= 
1511057400;
dateTime|inTemp|extraTemp3
1511057400|70.60002|75.02
1511057700|70.39998|74.66
1511058000|70.39998|74.66
1511058300|70.2767123287672|74.48
1511058600|70.2|
1511069400|69.8092307692308|



A f thought:  

In my driver if the correct data is not supplied, None is sent to 
event.record, is this correct, or should it simply not be called in that 
circumstance?
Snippit from my current driver:
for sensor in self.sensors:
if sensor != 'mac':
#print "DB Entry: " + self.sensors[sensor]
try:
value = esp_records[sensor]
except KeyError:
value = None

#print "Value: " + str(value)
event.record[self.sensors[sensor]] = value





I never modify the database without going through weewx, unless I am 
purging old data, so it shouldn't be an issue, but you never know.

Thanks Gary

On Saturday, November 18, 2017 at 8:39:03 PM UTC-7, gjr80 wrote:
>
> Yep, that could explain the symptoms. The accumulator for a given day and 
> for a given observation is at times initialised from that particular day 
> from that particular observation's daily summary table. If the sum field of 
> the day and daily summary concerned was None instead of a numeric then the 
> sum field of that accumulator would be initialised with None instead of 0.0 
> and you would see the symptoms you experienced. Rebuilding the daily 
> summaries would fix such a problem. It will be interesting to see if the 
> same issue comes back, certainly if you were making changes to the database 
> schema there is potential for a hiccup to occur, if you are relying on 
> weeWX to handle all database interraction then such a hiccup should never 
> occur.
>
> Gary
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing

[gjr80]

Yep, that could explain the symptoms. The accumulator for a given day and 
for a given observation is at times initialised from that particular day 
from that particular observation's daily summary table. If the sum field of 
the day and daily summary concerned was None instead of a numeric then the 
sum field of that accumulator would be initialised with None instead of 0.0 
and you would see the symptoms you experienced. Rebuilding the daily 
summaries would fix such a problem. It will be interesting to see if the 
same issue comes back, certainly if you were making changes to the database 
schema there is potential for a hiccup to occur, if you are relying on 
weeWX to handle all database interraction then such a hiccup should never 
occur.

Gary

On Sunday, 19 November 2017 11:23:03 UTC+10, Thomas Carlin wrote:
>
> I did a little more playing with this today, and determined that the 
> problem is with extraTemp3, which I just started using, but is built into 
> the system.  That being the case, I decided to drop and re-build the daily 
> summaries, just for fun.  My theory being that was done right before the 
> issue started occurring to add a battery voltage field, and there could 
> have been an issue there somehow.  It hasn't been running for too long yet, 
> but so far it seems to be working.  I'll keep an eye on it over the next 
> few days, and see how it does, and update the thread.
>
> Thanks,
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing

[Thomas Carlin]

I did a little more playing with this today, and determined that the 
problem is with extraTemp3, which I just started using, but is built into 
the system.  That being the case, I decided to drop and re-build the daily 
summaries, just for fun.  My theory being that was done right before the 
issue started occurring to add a battery voltage field, and there could 
have been an issue there somehow.  It hasn't been running for too long yet, 
but so far it seems to be working.  I'll keep an eye on it over the next 
few days, and see how it does, and update the thread.

Thanks,

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Weewx crashing

[Thomas Carlin]

Hey Gary,

The config file is below.  I didn't think about it yesterday, but I am 
using the history generator script (found here 
).
 
I have modified the schema to include a few extra data sources, but if i 
understand correctly, I have not tried to add any accumulators.  What would 
that code like for the initialization you mention?  (Other than = None)?  
Could I add a debug line somewhere to get a little more information out of 
the system to determine more precisely what accumulation is incorrect?  As 
a side note, since it started running again, it has run without hiccup for 
a day, with no changes at all.

Thanks again!

Datasources:
schema_with_esp = schemas.wview.schema + [('DoorOpen1', 'REAL'), (
'DoorClosed1', 'REAL'), ('DoorOpen2', 'REAL'), ('DoorClosed2', 'REAL'), (
'IrrigationFlow', 'REAL'), ('IrrigationPressure', 'REAL'),('esp1Signal', 
'REAL'), ('esp2Signal', 'REAL'),  ('esp3Signal', 'REAL'), ('esp3LIPOVoltage'
, 'REAL'), ('esp3LIPOCurrent', 'REAL'), ('esp3SolarVoltage', 'REAL'), (
'esp3SolarCurrent', 'REAL'), ('esp3OutputVoltage', 'REAL'), (
'esp3OutputCurrent', 'REAL'), ('extraHumid3', 'REAL'), ('esp4Signal', 'REAL'
), ('esp4LIPOVoltage', 'REAL')]


I also have added these to, and created some simple groups for these, 
following the documentation, which works as expected.  


Weewx.conf (Edited for brevity)
# WEEWX CONFIGURATION FILE
#
# Copyright (c) 2009-2015 Tom Keffer
# See the file LICENSE.txt for your rights.

##

# This section is for general configuration information.

# Set to 1 for extra debug info, otherwise comment it out or set to zero
debug = 1

# Root directory of the weewx data file hierarchy for this station
WEEWX_ROOT = /

# How long to wait before timing out a socket (FTP, HTTP) connection
socket_timeout = 20

# Do not modify this. It is used when installing and updating weewx.
version = 3.7.1

##

#   This section is for information about the additional ESP8266 Sensors 
that are in use.

[ESP8266]
[[http://1.2.3.4]]
mac = 5C:CF:7F:02:0C:17
Temp = extraTemp1
Humid = extraHumid1
doorOpen = DoorOpen1
doorClosed = DoorClosed1
Signal = esp1Signal

[[http://1.2.3.4]]
mac = 5C:CF:7F:07:5B:90
Temp = extraTemp2
Humid = extraHumid2
doorOpen = DoorOpen2
doorClosed = DoorClosed2
Signal = esp2Signal

#[[http://1.2.3.4]]
#mac = 30:AE:A4:30:93:54
#Temp = extraTemp3
#Humid = extraHumid3
# Signal = esp4Signal
# Voltage = esp4LIPOVoltage
#[[http://1.2.3.4]]
#mac = 5C:CF:7F:12:17:97
#Flow = IrrigationFlow
#Pressure = IrrigationPressure
# Signal = esp3Signal
# LIPO_Voltage = esp3LIPOVoltage
# LIPO_Current = esp3LIPOCurrent
# Solar_Voltage = esp3SolarVoltage
# Solar_Current = esp3SolarCurrent
# Output_Voltage = esp3OutputVoltage
# Output_Current = esp3OutputCurrent

##

#   This section is for information about the station.

[Station]

# Description of the station location
location = y

# Latitude and longitude in decimal degrees
latitude = 
longitude = 

# Altitude of the station, with unit it is in. This is downloaded from
# from the station if the hardware supports it.
altitude = 1234, foot

# Set to type of station hardware. There must be a corresponding stanza
# in this file with a 'driver' parameter indicating the driver to be 
used.
station_type = Vantage

# If you have a website, you may specify an URL
#station_url = http://www.example.com

# The start of the rain year (1=January; 10=October, etc.). This is
# downloaded from the station if the hardware supports it.
rain_year_start = 1

# Start of week (0=Monday, 6=Sunday)
week_start = 6

##

[Vantage]
# This section is for the Davis Vantage series of weather stations.

# Connection type: serial or ethernet 
#  serial (the classic VantagePro)
#  ethernet (the WeatherLinkIP)
type = ethernet

# If the connection type is serial, a port must be specified:
#   Debian, Ubuntu, Redhat, Fedora, and SuSE:
# /dev/ttyUSB0 is a common USB port name
# /dev/ttyS0   is a common serial port name
#   BSD:
# /dev/cuaU0   is a common serial port name
port = /dev/ttyUSB0

# If the connection type is ethernet, an IP Address/hostname is 
required:
host = 1.2.3.4

##
# The rest of this section rarely needs any attention. 
# You can safely leave it "as is."

[weewx-user] Re: Weewx crashing

[Thomas Carlin]

Sorry, Debian 9.2, Stretch, fully updated.  I misread /proc/version


On Thursday, November 16, 2017 at 8:56:28 PM UTC-7, Thomas Carlin wrote:
>
> Hey Tom et al.,
>
> I have an issue with Weewx, which has been running well until yesterday.  
> I am running it on Debian 6.3.0-18, fully up to date with a Vantage Pro 2.  
> Weewx was installed through apt-get, and is fully up to date.  I do have 
> several modifications, including adation sensors that I wrote a small 
> driver for, but they have not caused me any issues in the past, and in the 
> event that a sensor is unavailable, fail gracefully.  I have not modified 
> any 'original' scripts, but have added fields to the database, tags, the 
> mentioned driver, and am using the cmon plugin.  I would be happy to 
> provide more details if necessary.
>
> The error in syslog is below.  Please note that this is the first 
> occurrence of this (that I can find in my logs).
>
> Nov 15 19:30:20 Ds-1 weewx[27964]: manager: Added record 2017-11-15 
> 19:30:20 MST (1510799420) to database 'cmon.sdb'
> Nov 15 19:30:20 Ds-1 weewx[27964]: manager: Added record 2017-11-15 
> 19:30:20 MST (1510799420) to daily summary in 'cmon.sdb'
> Nov 15 19:30:20 Ds-1 weewx[27964]: manager: Added record 2017-11-15 
> 19:30:00 MST (1510799400) to database 'weewx.sdb'
> Nov 15 19:30:20 Ds-1 weewx[27964]: engine: Main loop exiting. Shutting 
> engine down.
> Nov 15 19:30:20 Ds-1 weewx[27964]: engine: Shutting down StdReport thread
> Nov 15 19:30:20 Ds-1 weewx[27964]: engine: StdReport thread has been 
> terminated
> Nov 15 19:30:21 Ds-1 weewx[27964]: engine: Caught unrecoverable exception 
> in engine:
> Nov 15 19:30:21 Ds-1 weewx[27964]:   unsupported operand type(s) 
> for +=: 'NoneType' and 'float'
> Nov 15 19:30:21 Ds-1 weewx[27964]:   Traceback (most recent call 
> last):
> Nov 15 19:30:21 Ds-1 weewx[27964]: File 
> "/usr/share/weewx/weewx/engine.py", line 871, in main
> Nov 15 19:30:21 Ds-1 weewx[27964]:   engine.run()
> Nov 15 19:30:21 Ds-1 weewx[27964]: File 
> "/usr/share/weewx/weewx/engine.py", line 201, in run
> Nov 15 19:30:21 Ds-1 weewx[27964]:   
> self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
> Nov 15 19:30:21 Ds-1 weewx[27964]: File 
> "/usr/share/weewx/weewx/engine.py", line 223, in dispatchEvent
> Nov 15 19:30:21 Ds-1 weewx[27964]:   callback(event)
> Nov 15 19:30:21 Ds-1 weewx[27964]: File 
> "/usr/share/weewx/weewx/engine.py", line 561, in post_loop
> Nov 15 19:30:21 Ds-1 weewx[27964]:   self._software_catchup()
> Nov 15 19:30:21 Ds-1 weewx[27964]: File 
> "/usr/share/weewx/weewx/engine.py", line 633, in _software_catchup
> Nov 15 19:30:21 Ds-1 weewx[27964]:   
> self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD, 
> record=record, origin='software'))
> Nov 15 19:30:21 Ds-1 weewx[27964]: File 
> "/usr/share/weewx/weewx/engine.py", line 223, in dispatchEvent
> Nov 15 19:30:21 Ds-1 weewx[27964]:   callback(event)
> Nov 15 19:30:21 Ds-1 weewx[27964]: File 
> "/usr/share/weewx/weewx/engine.py", line 588, in new_archive_record
> Nov 15 19:30:21 Ds-1 weewx[27964]:   
> dbmanager.addRecord(event.record, accumulator=self.old_accumulator)
> Nov 15 19:30:21 Ds-1 weewx[27964]: File 
> "/usr/share/weewx/weewx/manager.py", line 245, in addRecord
> Nov 15 19:30:21 Ds-1 weewx[27964]:   
> self._addSingleRecord(record, cursor, log_level)
> Nov 15 19:30:21 Ds-1 weewx[27964]: File 
> "/usr/share/weewx/weewx/manager.py", line 1183, in _addSingleRecord
> Nov 15 19:30:21 Ds-1 weewx[27964]:   
> _day_summary.addRecord(record, weight=_weight)
> Nov 15 19:30:21 Ds-1 weewx[27964]: File 
> "/usr/share/weewx/weewx/accum.py", line 242, in addRecord
> Nov 15 19:30:21 Ds-1 weewx[27964]:   func(self, record, 
> obs_type, add_hilo, weight)
> Nov 15 19:30:21 Ds-1 weewx[27964]: File 
> "/usr/share/weewx/weewx/accum.py", line 301, in add_value
> Nov 15 19:30:21 Ds-1 weewx[27964]:   
> self[obs_type].addSum(val, weight=weight)
> Nov 15 19:30:21 Ds-1 weewx[27964]: File 
> "/usr/share/weewx/weewx/accum.py", line 88, in addSum
> Nov 15 19:30:21 Ds-1 weewx[27964]:   self.sum += val
> Nov 15 19:30:21 Ds-1 weewx[27964]:   TypeError: unsupported 
> operand type(s) for +=: 'NoneType' and 'float'
> Nov 15 19:30:21 Ds-1 weewx[27964]:   Exiting.
>
>
> I do have a watchdog on the process that tries to re-launch weewx if it 
> detects that it is not running.  Most recently it crashed at 11:00 A, and 
> the watchdog tried to re-launch every minute until 19:30, when it started 
> successfully, and the system threw the same error every time.  
>
> Do you need any more information from me, and/or is there something else 
> that I can do to help troubleshoot this issue?
>
> Thank you in a