Thank you :) so when I said:
"For backfilling missing data would use an approach like: If there is data available in weatherflow REST service, an if it is available on a time base which is more frequent than archive_interval, I would try to backfill like this: for every top of an weewx archive_interval that is missing, I'd get all data available for this interval, calculate averages (take care with vector data windSpeed/windDir!), check if minimum/maximum values of each REST entry, if it is lower/higher than the min/max value in archive_day_[archiveColumn], handle it." I'd now say: For a weatherflow driver I wouldn't set an archive_interval in the driver itself. For backfilling, get the all records from the REST within the current archive_interval to backfill, create a single record for that interval, with the values being the average values of all the REST records (with the right treatment for vector data). Losing precision of min/max values and their timestamps is acceptable (at least for reasonable short archive_interval settings) gjr80 schrieb am Dienstag, 9. Februar 2021 um 13:48:18 UTC+1: > Very simply, when your driver is emitting loop packets your time > resolution is the time between loop packet timestamps, once your driver is > emitting archive records your time resolution decreases and is the time > between archive record timestamps. Similar but slightly different with > actual values, loop packets give you the finest granularity of the > observation values, once you move to archive records you get the average of > the loop values (well actually you get whatever function is used to obtain > an archive record value from a group of loop packet values, this is usually > the average but it could be the sum or the last or the first value, it > depends). > > Gary > > On Tuesday, 9 February 2021 at 22:42:18 UTC+10 gjr80 wrote: > >> OK, let's work through it. The hardware stored data at one minute >> intervals, so between 11:05:00am to 11:10:00am the hardware will have five >> stored records, the driver then creates a single archive record covering >> 11:05:00am to 11:10:00am, the observation that has the max value at >> 11:07:00am will appear as the average of the five values stored between >> 11:05:00am and 11:10:00am (so chances are that will not be the same as the >> value (the maximum) seen at 11:07:00am) and that is what will be stored as >> the new daily max in the daily summary (if in fact it is a new maximum). >> The timestamp of this backfilled record will be 11:10:00am and that is what >> will be recorded as the max timestamp in the daily summary. So in this case >> not only will you likely miss the maximum value that occurred at 11:07:00am >> but you will miss the actual timestamp of the maximum value as well, it >> will be moved to an archive interval boundary. >> >> Gary >> >> On Tuesday, 9 February 2021 at 22:28:52 UTC+10 michael.k...@gmx.at wrote: >> >>> For the loop packets this is what I already thought to know. >>> >>> What I wanted to know if my weew archive interval is 5 mins, and the >>> hardware stored at 1min: >>> Lets assume the archive_interval 11:05:00am to 11:10:00am is to be >>> backfilled. The driver gets the stored archive values from the hardware, >>> where the maxvalue was at 11:07:00am. >>> Is the maxvalue in the archive_day_xxx table for that day timestamped >>> with 11:07:00am or with 11:10:00am? >>> >>> gjr80 schrieb am Dienstag, 9. Februar 2021 um 13:11:41 UTC+1: >>> >>>> If your one loop packet was the only loop packet received by WeeWX in >>>> that archive interval then the archive record that WeeWX would synthesise >>>> from that solitary loop packet would be the same as that loop packet. If >>>> as >>>> you said that loop packet was stored by hardware and if the driver's >>>> genArchiveRecord() method obtains that same data when it generates an >>>> archive record, then yes they will be the same. Highs and lows in the >>>> daily >>>> summaries would be the same but it would not surprise me if some >>>> timestamps >>>> were different. Loop packets are generally timestamped when the data is >>>> read, whereas archive records are typically timestamped at the end of the >>>> period they cover. So for an archive interval that covers say 11:05:00am >>>> to >>>> 11:06:00am, the loop packet may have been created at 11:05:25am and >>>> timestamped accordingly. If WeeWX is set to obtain highs and lows from >>>> loop >>>> packets (ie loop_hilo <http://weewx.com/docs/usersguide.htm#StdArchive> >>>> is True) then if a new max/min is found in that loop packet the relevant >>>> timestamp recorded in the daily summary will be the loop packet timestamp. >>>> If you now consider the archive record that is read during a backfill, it >>>> will be timestamped 11:06:00am, again any new highs or lows will be >>>> recorded (so the actual aggregate value will be the same) but in this case >>>> the timestamp recorded for the high or low will be the archive record >>>> timestamp 11:06:00am. If you happen to have more than one loop packet in >>>> the archive period then you may loose some fidelity of the actual >>>> observation high/low value as well. >>>> >>>> This is the same effect you see if you rebuild your daily summaries >>>> with wee_database --rebuild-daily, if you have loop_hilo = True then >>>> your daily summary tables will likely have loop packet timestamps for many >>>> highs and lows but after you rebuild the daily summaries you loose that >>>> fine detail and your high/low timestamps will align with archive record >>>> timestamps. If you have multiple loop packets in an archive interval then >>>> your high/low values may be one of the values in those loop packets, after >>>> rebuilding though you end up with archive values being used to generate >>>> the >>>> highs and lows which are typically averages of loop values. For example, >>>> if >>>> you have four loop packets in an archive interval and each has an >>>> outTemp value of 8.0, 9.0, 12.0 and 7.0 degrees C your daily summary >>>> may record 12.0 as the max. The corresponding archive record outTemp >>>> value would (8.0+9.0+12.0+7.0)/4=9.0 so your max might be 9.0. >>>> >>>> Gary >>>> >>>> On Tuesday, 9 February 2021 at 21:03:10 UTC+10 michael.k...@gmx.at >>>> wrote: >>>> >>>>> > If you mean does WeeWX determine if any incoming archive record has >>>>> (for example) a new daily max for some field and then save that data, >>>>> then >>>>> yes that is what StdArchive does. >>>>> >>>>> So if a hardware sends a loop packet with all the readings in a >>>>> one-minute-interval, and stores these exact values in its own storage in >>>>> a >>>>> separate entry, one entry per minute, the weewx database values are >>>>> exactly >>>>> the same, regardless if they were received as loop data as if they were >>>>> backfilled? Including timestamps and Values for min/max in >>>>> archive_day_xxxx, which may differ from the min/max value found in all >>>>> weewx archive records of that particular day, with timestamps between >>>>> possible timestamps of archve values? >>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/1f5feebf-d243-458d-b1dd-5b172f4e5403n%40googlegroups.com.
