I'm sure you are right about the triggering circumstances. The code in my
weewx User routines was correct (using the Meteo France limit), but I had
put in some modifications around the hauteur_soleil test in connection with
the method I am using to average and 'normalize' the observations for the
archive reports. I suspect that, in taking out those modifications for the
archive update code, I inadvertently also removed the original test.
On Friday, July 1, 2022 at 10:47:43 AM UTC-4 jterr...@gmail.com wrote:
> OK.
> The variable "hauteur_soleil is the elevation of the sun, in degree.
> From sunset to sunrise, this value will be negative (i.e. sun below the
> horizon).
> The formula developed by Metro France was validated to be effective only
> if sun elevation is > 3°.
>
> In my own implantation of this algorithm, I decided to start the
> calculation of the sun threshold as soon as the sun elevation was > 0°
> *and* that the global radiation as measured by the Davis pyranometer was
> greeter than 20 W/m2.
>
> I suspect that the few dateTime that trigger your errors were in a
> situation in which sun elevation was just below zero and that the measured
> radiation was just higher than 20 W/m2
>
> Le 1 juil. 2022 à 16:19, 'Peter Fletcher' via weewx-user <
> weewx...@googlegroups.com> a écrit :
>
> Something like that is obviously needed. I don't know how my copy of the
> code omitted the test for hauteur_soleil being > 0. There are a number of
> sources from which I may have copied it, and I didn't keep links to all of
> them, but I can't imagine why I would have deleted the test if my source
> had included it. I will obviously add it to my working code.
>
> On Friday, July 1, 2022 at 2:45:42 AM UTC-4 jterr...@gmail.com wrote:
>
>> OK. What I don't understand is that my code had always a test
>>
>> if hauteur_soleil > 3: (according to Météo France)
>>
>> or more recently
>>
>> if hauteur_soleil > 0:
>>
>> in the function, and this test is not on your function.
>>
>> It should be :
>>
>> hauteur_soleil = asin(sin((pi / 180) * latitude) * sin((pi / 180) *
>> declinaison) + cos(
>> (pi / 180) * latitude) * cos((pi / 180) * declinaison) * cos((pi
>> / 180) * angle_horaire)) * (180 / pi)
>> If hauteur_soleil > 0:
>>
>> seuil = (0.73 + 0.06 * cos((pi / 180) * 360 * dayofyear / 365)) * 1080
>> * pow(
>> (sin(pi / 180) * hauteur_soleil), 1.25) * coeff
>> else :
>> seuil = 0
>> return seuil
>>
>> Le 30 juin 2022 à 22:52, 'Peter Fletcher' via weewx-user <
>> weewx...@googlegroups.com> a écrit :
>>
>> That was going to be my next step! In fact, iterating through a list of
>> the dateTime values that produce the errors in the real code and passing
>> each value to the function confirms that it is the specific dateTime values
>> that are causing the function to misbehave. The returned results are all
>> complex numbers with negative and numerically identical (for a given
>> dateTime) real and imaginary components. It does seem to be a bug in the
>> function. I assume that hauteur_soleil should always be >=0. It appears
>> that, for my latitude and longitude and for the given specific values of
>> dateTime, it becomes negative. The last step in the calculation then
>> involves raising a negative number to a non-integral power, which is
>> guaranteed to produce interesting results! The really odd thing is that
>> math.pow is not returning a ValueError, which the docs say is what should
>> happen under these circumstances, but apparently trying to return a
>> (possibly) valid complex result.
>> On Thursday, June 30, 2022 at 3:07:38 PM UTC-4 jterr...@gmail.com wrote:
>>
>>> The only clue I have is that the problem is not due to an
>>> « overloading » of your raspberry pi, but seems to occur with specific
>>> dateTime values.
>>> You can try to run your script only with a « bad » dateTime :
>>>
>>> "SELECT dateTime, Radiation from archive where dateTime = 1592614500 »
>>>
>>> Does the error occurs ? If yes, you can try to add debugging print
>>> commands inside the sunshineThreshold function to try to understand.
>>>
>>>
>>>
>>> Le 30 juin 2022 à 19:51, 'Peter Fletcher' via weewx-user <weewx...@
>>> googlegroups.com> a écrit :
>>>
>>> It did as it seems you predicted - passed 1592614800 and stopped at
>>> 1632611100. You obviously have a clue as to what is going on. Please
>>> explain!
>>>
>>> On Thursday, June 30, 2022 at 8:59:48 AM UTC-4 jterr...@gmail.com wrote:
>>>
>>>> If you exclude the first one,1592614500 , with a query like "SELECT
>>>> dateTime, Radiation from archive where dateTime <> 1592614500", will the
>>>> script stop at 1592614800 ( the next dateTime) or will it continue and
>>>> stop
>>>> at 1632611100 ?
>>>>
>>>> Le 30 juin 2022 à 14:34, 'Peter Fletcher' via weewx-user <weewx...@
>>>> googlegroups.com> a écrit :
>>>>
>>>> 1592614500
>>>> 1632611100
>>>> 1632611400
>>>> 1647688800
>>>>
>>>> I can't see a pattern or any common features.
>>>>
>>>> On Thursday, June 30, 2022 at 3:55:49 AM UTC-4 jterr...@gmail.com
>>>> wrote:
>>>>
>>>>> No, I never had weewx crashes related to the sunshine calculations.
>>>>>
>>>>> What are the dateTime values that trigger the error ?
>>>>>
>>>>>
>>>>>
>>>>> Le mercredi 29 juin 2022 à 23:23:16 UTC+2, Peter Fletcher a écrit :
>>>>>
>>>>>> Have you had any odd weewx errors or crashes related to the sunshine
>>>>>> calculations? I ask because I hadn't, but I decided to try to 'backfill'
>>>>>> my
>>>>>> database with sunshine times, based on the 5-minute radiation values,
>>>>>> and I
>>>>>> ran into a bizarre bug. I used the code shown below (on a copy of my
>>>>>> live
>>>>>> weewx database). As you will see, the threshold calculation code is
>>>>>> essentially identical to yours, except that it has been converted to a
>>>>>> regular function (no 'self' parameter) and my station's latitude and
>>>>>> longitude are hard coded in it. When the code is run under Python 3.9.2
>>>>>> on
>>>>>> my Pi, it initially runs without problems, but crashes after 8,000+
>>>>>> records
>>>>>> have been processed with a ValueError on the MaxThreshold vs threshold
>>>>>> comparison, reporting that it can't compare a complex with a float! If I
>>>>>> intercept and log the errors, it turns out that, for a few specific
>>>>>> values
>>>>>> of dateTime, the function returns a complex number! Even more bizarrely,
>>>>>> it
>>>>>> only seems to do that in the context of the running code. If I manually
>>>>>> run
>>>>>> through all the operations from the function code at the Python command
>>>>>> line, using the value of dateTime that produces the first crash, all the
>>>>>> intermediate results and the final result are sane floats.
>>>>>> There appears to be a second issue, possibly related to my reading
>>>>>> and writing the database at relatively high frequency, which stalls the
>>>>>> process after about 18,000 records have been processed, but removing the
>>>>>> database writes allows it to run to completion without abolishing the
>>>>>> consistent, albeit infrequent, ValueErrors.
>>>>>>
>>>>>> [backfill.py]
>>>>>> import sqlite3
>>>>>> from datetime import datetime
>>>>>> import time
>>>>>> from math import sin, cos, pi, asin
>>>>>>
>>>>>> def sunshineThreshold(mydatetime):
>>>>>> coeff = 0.9 # change to calibrate with your sensor
>>>>>> utcdate = datetime.utcfromtimestamp(mydatetime)
>>>>>> dayofyear = int(time.strftime("%j", time.gmtime(mydatetime)))
>>>>>> theta = 360 * dayofyear / 365
>>>>>> equatemps = 0.0172 + 0.4281 * cos((pi / 180) * theta) - 7.3515 *
>>>>>> sin(
>>>>>> (pi / 180) * theta) - 3.3495 * cos(2 * (pi / 180) * theta) -
>>>>>> 9.3619 * sin(
>>>>>> 2 * (pi / 180) * theta)
>>>>>>
>>>>>> latitude = 43.0346213
>>>>>> longitude = -78.689362
>>>>>>
>>>>>> corrtemps = longitude * 4
>>>>>> declinaison = asin(0.006918 - 0.399912 * cos((pi / 180) * theta)
>>>>>> + 0.070257 * sin(
>>>>>> (pi / 180) * theta) - 0.006758 * cos(2 * (pi / 180) * theta)
>>>>>> + 0.000908 * sin(
>>>>>> 2 * (pi / 180) * theta)) * (180 / pi)
>>>>>> minutesjour = utcdate.hour * 60 + utcdate.minute
>>>>>> tempsolaire = (minutesjour + corrtemps + equatemps) / 60
>>>>>> angle_horaire = (tempsolaire - 12) * 15
>>>>>> hauteur_soleil = asin(sin((pi / 180) * latitude) * sin((pi /
>>>>>> 180) * declinaison) + cos(
>>>>>> (pi / 180) * latitude) * cos((pi / 180) * declinaison) *
>>>>>> cos((pi / 180) * angle_horaire)) * (180 / pi)
>>>>>> seuil = (0.73 + 0.06 * cos((pi / 180) * 360 * dayofyear / 365))
>>>>>> * 1080 * pow(
>>>>>> (sin(pi / 180) * hauteur_soleil), 1.25) * coeff
>>>>>> return seuil
>>>>>>
>>>>>>
>>>>>> database = 'weewx.sdb'
>>>>>>
>>>>>> maxThreshold=0
>>>>>> count=0
>>>>>> conn=sqlite3.connect(database)
>>>>>> cur=conn.execute("SELECT dateTime, Radiation from archive")
>>>>>> for row in cur:
>>>>>> count += 1
>>>>>> if (row[1] is not None) and (row[1] > 20):
>>>>>> threshold = sunshineThreshold(row[0])
>>>>>> if threshold > maxThreshold:
>>>>>> maxThreshold = threshold
>>>>>> if row[1] > threshold:
>>>>>> conn.execute("UPDATE archive set SunshineTime = 5 WHERE
>>>>>> dateTime = " + str(row[0]))
>>>>>> if count % 1000 == 0:
>>>>>> print(count, 'Max Threshold', maxThreshold)
>>>>>> conn.close
>>>>>> [/backfill.py]
>>>>>>
>>>>>> On Friday, June 10, 2022 at 3:29:40 AM UTC-4 jterr...@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>> On my side, I have looked at the CPU utilization on my raspberry Pi
>>>>>>> 3B+. I have the mqtt service service installed, so at each loop all
>>>>>>> data
>>>>>>> of the packet are sent to the mqtt broker.
>>>>>>>
>>>>>>> With mqtt and when calculations of the sunshine threshold is done
>>>>>>> for each loop packet, the total CPU utilization of python3 is about
>>>>>>> 0.75%
>>>>>>> With mqtt and without calculation of sunshine threshold : 0.5% of
>>>>>>> total CPU.
>>>>>>>
>>>>>>> So one can estimate that 0.25 % of total CPU is needed for the
>>>>>>> calculation of the threshold value for each LOOP packet.
>>>>>>>
>>>>>>>
>>>>>>> Le 9 juin 2022 à 22:26, 'Peter Fletcher' via weewx-user <weewx...@
>>>>>>> googlegroups.com> a écrit :
>>>>>>>
>>>>>>> After some experimentation, I found that the radiation value in the
>>>>>>> VP2 LOOP packets does, indeed, normally change every 50-52 seconds,
>>>>>>> but,
>>>>>>> perhaps about a fifth of the 'gaps' are a *multiple* of that time -
>>>>>>> most often 100+ or 150+ seconds, but occasionally more than that (I saw
>>>>>>> one
>>>>>>> 250+ second 'gap'). I saw this under conditions of variable sunshine
>>>>>>> and
>>>>>>> clouds when it seemed unlikely that the actual radiation value would
>>>>>>> have
>>>>>>> been precisely constant for that length of time, so I am not sure
>>>>>>> exactly
>>>>>>> what is going on. In any event, I am revising the code I am using on
>>>>>>> the
>>>>>>> basis of doing the threshold calculation when the radiation level
>>>>>>> changes,
>>>>>>> but at least every minute, if it remains constant for more than the
>>>>>>> normal
>>>>>>> 50-52 seconds..
>>>>>>>
>>>>>>> On Sunday, June 5, 2022 at 12:33:47 PM UTC-4 jterr...@gmail.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I think it is also OK to do an average for every 30 seconds. It
>>>>>>>> depends also on the weather station used.
>>>>>>>> For instance, a Davis Vantage Pro 2 ISS transmits an
>>>>>>>> updated solar radiation value every 50 to 60 seconds. So with this
>>>>>>>> weather
>>>>>>>> station, even a 1 minute average would not be very different since
>>>>>>>> anyway
>>>>>>>> the solar radiation values of the LOOP packet are the same for at
>>>>>>>> least 50
>>>>>>>> seconds.!
>>>>>>>>
>>>>>>>> Le 5 juin 2022 à 18:02, 'Peter Fletcher' via weewx-user <weewx...@
>>>>>>>> googlegroups.com> a écrit :
>>>>>>>>
>>>>>>>> I chose to average the LOOP radiation readings and only to do the
>>>>>>>> threshold calculation and make the sun/no sun determination every 30
>>>>>>>> seconds because I thought doing it on every LOOP might overload LOOP
>>>>>>>> processing (I am running weewx on a Pi 3B, which is also doing a few
>>>>>>>> other
>>>>>>>> things which use the CPU). If this is an unnecessary concern, as it
>>>>>>>> may
>>>>>>>> very well be, your modified code is much cleaner than mine.
>>>>>>>>
>>>>>>>> On Saturday, June 4, 2022 at 12:41:08 PM UTC-4 jterr...@gmail.com
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> It is a very good idea to calculate the sunshine duration for each
>>>>>>>>> LOOP packet and sum these values to make the final archive sunshine
>>>>>>>>> duration. I have modified my script accordingly :
>>>>>>>>> https://github.com/Jterrettaz/sunduration.
>>>>>>>>> The logic is the following : for each received LOOP packet, the
>>>>>>>>> radiation is compared to a calculated threshold. If the radiation is
>>>>>>>>> above
>>>>>>>>> the threshold value, the sunshine time for the LOOP packet is equal
>>>>>>>>> to the
>>>>>>>>> time elapsed between the previous loop packet and this packet (most
>>>>>>>>> of the
>>>>>>>>> time 2 seconds with a Vantage Davis Pro).
>>>>>>>>> The final archive sunshine duration is the sum of all the LOOP
>>>>>>>>> value within the archive period.
>>>>>>>>> Le vendredi 3 juin 2022 à 21:59:36 UTC+2, Peter Fletcher a écrit :
>>>>>>>>>
>>>>>>>>>> That makes some sense when you are getting data from an
>>>>>>>>>> 'external' sensor, though there are (IMHO) simpler ways of doing it.
>>>>>>>>>> weewx
>>>>>>>>>> already has access to the LOOP radiation data from the VP2, so
>>>>>>>>>> handling the
>>>>>>>>>> processing and data storage within weewx makes more sense to me in
>>>>>>>>>> this
>>>>>>>>>> case.
>>>>>>>>>>
>>>>>>>>>> On Friday, June 3, 2022 at 3:24:23 PM UTC-4 vince wrote:
>>>>>>>>>>
>>>>>>>>>>> On Friday, June 3, 2022 at 11:17:00 AM UTC-7 Meteo Oberwallis
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> if the interval of Weewx and the data logger is set to 10
>>>>>>>>>>>> minutes, I would have liked to read the value of the solar sensor
>>>>>>>>>>>> every
>>>>>>>>>>>> minute and then write it into a separate .sdb database as possible
>>>>>>>>>>>> sunshine.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Personally I'd use an external program called via cron and
>>>>>>>>>>> posting a message to a MQTT topic. Have weewx subscribe to that
>>>>>>>>>>> topic to
>>>>>>>>>>> get the data into your db.
>>>>>>>>>>>
>>>>>>>>>>> This is how I used to get my DS18b20 temperature sensor data
>>>>>>>>>>> into weewx.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>> the Google Groups "weewx-user" group.
>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe
>>>>>>>> .
>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>> weewx-user+...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/weewx-user/0e631671-0a74-4963-9f1c-e5f81bc7c366n%40googlegroups.com
>>>>>>>>
>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/0e631671-0a74-4963-9f1c-e5f81bc7c366n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>> the Google Groups "weewx-user" group.
>>>>>>> To unsubscribe from this topic, visit
>>>>>>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe
>>>>>>> .
>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>> weewx-user+...@googlegroups.com.
>>>>>>>
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/weewx-user/f0ecc86f-a615-4a24-a43f-ee0d3963b8adn%40googlegroups.com
>>>>>>>
>>>>>>> <https://groups.google.com/d/msgid/weewx-user/f0ecc86f-a615-4a24-a43f-ee0d3963b8adn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>>>
>>>>>>>
>>>> --
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "weewx-user" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> weewx-user+...@googlegroups.com.
>>>>
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/weewx-user/39cf6daa-80ca-4ffb-89d3-0f00b971481an%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/weewx-user/39cf6daa-80ca-4ffb-89d3-0f00b971481an%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>>
>>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "weewx-user" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> weewx-user+...@googlegroups.com.
>>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-user/c07ed2bb-1e3d-43e2-b08c-08a7a3aa92dbn%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/weewx-user/c07ed2bb-1e3d-43e2-b08c-08a7a3aa92dbn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "weewx-user" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> weewx-user+...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/weewx-user/f287c1b3-1005-409c-82a9-a072e375d5e9n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/weewx-user/f287c1b3-1005-409c-82a9-a072e375d5e9n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/weewx-user/19ylVTRqbh4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> weewx-user+...@googlegroups.com.
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/77cd4de6-3b1c-4bc2-9dbc-8d8316bd8a9bn%40googlegroups.com
>
> <https://groups.google.com/d/msgid/weewx-user/77cd4de6-3b1c-4bc2-9dbc-8d8316bd8a9bn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/18425607-f8e0-4708-8344-1c1df95ae8b2n%40googlegroups.com.
