Tom,
Your interpretation is exactly correct, and yes it is a coding error on my
side. My thought was just a single sentence there about the Weewx engine
not doing any sanitation, and that being the responsibility of the driver.
If not, no worries, just a thought.
Gary,
No worries on the delay. You got me going in the right direction, and we
got it figured out! It's not like you get paid for doing this. Until
yesterday, the extent of the sanitation that I was doing on the incoming
data was running float(data), and catching any ValueError exception. Now,
because nan is a special 'string', when you run float('nan'), it returns
NaN, but now it is the Python 'number' NaN. When you add a number to NaN,
your result is always NaN. When this is stuffed into the database,
apparently it is interpreted as SQL NULL. Next time the loop runs, Python
queries the database, and interprets SQL NULL as None, and then we get the
type mismatch.
I have added some more sanitation for this, and we are back up and running,
and it *should* conform to expected behavior.
Does that make sense?
Thank you for the input on the driver. I have a laundry list of things
that I would like to add to make it more robust, and I'll add the loop
output to that. I may be converting to a queue as well, simply due to the
number of sensors that are being polled, and the "couple of seconds" note
in the Second Datasource section of the docs. But that is a project for
another day!
On Mon, Nov 20, 2017 at 12:53 AM, gjr80 <gjroder...@gmail.com> wrote:
> Thomas C,
>
> Sorry I did not respond to your earlier posts after mine, the extract you
> provided does indeed show a daily summary record with sum and wsum fields
> as None (well the db fields contain nothing but they will end up as
> python None values within weeWX). At the moment I have still not been
> able to fathom the circumstances that would cause this.
>
> One thing about your driver though. If you are using the default weeWX
> schema, or a modified version of the default schema to which you have only
> added numeric (read SQL type REAL) fields, then your driver must only emit
> numeric values or the python None value. If you emit strings it will
> cause an exception to be thrown within weeWX. The default weeWX behaviour
> is if the sensor is present and emitting values then the driver should emit
> those values as numeric values in the approriate loop packet/archive record
> field, if the sensor is present but offline then the driver emits the
> python None value in the field concerned and if the sensor does not exist
> then the driver should not emit anything for that sensor ie the field is
> excluded from the loop packet/archive record.
>
> Note. Technically there is no reason that you cannot add a string to a
> loop packet or an archive record but if it is placed in a field that weeWX
> tries to 'accumulate' it will cause an exception.
>
> I still don't see how your driver can cause the exception you first posted
> (emitting a string will cause a similar but clearly different exception),
> but it may be worthwhile reworking your driver to conform with the weeWX
> expected behaviour. It might also be worthwhile instrumenting your driver
> to drop the emitted loop packet to the log so we can see exactly what is
> coming out of the driver in the leadup to the exception.
>
> Gary
>
>
> On Monday, 20 November 2017 08:33:28 UTC+10, Thomas Carlin wrote:
>>
>> Hey Tom
>>
>> At this point in the code 'nan' is not equivalent to 'Not a Number', it
>> is simply a string that is pulled out of an http page, not generated by
>> python. Originally, it is generated on my Arduino when data cannot be
>> pulled from the temperature sensor. Unfortunately in this case when you
>> run float("nan") in python, it returns nan, and that was the extent of my
>> sanitization. I'll write a condition into my driver to catch that, and
>> should be good to go.
>>
>> On Sunday, November 19, 2017 at 2:14:59 PM UTC-7, Tom Keffer wrote:
>>>
>>> But, how did you create a NaN? They are not returned from any of the
>>> Python mathematical functions.
>>>
>>> -tk
>>>
>>> On Sun, Nov 19, 2017 at 2:04 PM, Thomas Carlin <thomas...@gmail.com>
>>> wrote:
>>>
>>>> I'm pretty sure this is what is happening. Take a look at the output
>>>> below:
>>>>
>>>> ...
>>>> DB Entry: extraTemp3
>>>> Value: nan
>>>> DB Entry: extraHumid3
>>>> Value: nan
>>>> ...
>>>> Unable to access http://192.168.2.105
>>>> REC: 2017-11-19 12:38:00 MST (1511120280) ..., *extraHumid3: nan*,
>>>> extraTemp1: 55.58, *extraTemp3: nan*, heatindex: 31.7143461695...
>>>>
>>>> The next attempt to add data to the database resulted in the program
>>>> crashing due to a type mismatch.
>>>>
>>>> May I request that a blurb be added to the documentation here:
>>>> http://www.weewx.com/docs/customizing.htm#Adding_2nd_source about
>>>> sanitizing the data before it is fed to event.record()?
>>>>
>>>>
>>>> On Sunday, November 19, 2017 at 10:59:29 AM UTC-7, Thomas Carlin wrote:
>>>>>
>>>>> The plot thickens. Right before I went to bed this morning, I checked
>>>>> the current daily summary record for extraTemp3, and it looked as you
>>>>> would
>>>>> expect it to. It had data in it for both sum and wsum, unlike the one
>>>>> posted above. Roughly an hour later, Weewx crashed with the same error,
>>>>> and this morning when I went to look into it, the current record, same one
>>>>> as last night, no longer had data in it for the same fields.
>>>>>
>>>>> After looking a little closer, I noticed that the extraHumid3, which I
>>>>> created, did exactly the same thing as extraTemp3. Since both these
>>>>> pieces
>>>>> of data come from the same DHT11 sensor, my current theory is that under
>>>>> the wrong conditions, my sensor is spitting out garbage that is not being
>>>>> properly sanitized, which somehow empties those fields. Then next time
>>>>> the
>>>>> loop runs, everything falls apart. I'm setting up a sandbox right now so
>>>>> I
>>>>> can play with it without screwing with my 'production' system. Just
>>>>> hoping
>>>>> the sandbox crashes like the current system does!
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "weewx-user" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to weewx-user+...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/weewx-user/TE57NMTaIXw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
Thomas Carlin
970-401-3805
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
