Dear Leon, Thanks for all your work on this. This works as well as we are going to be able to manage unless and until WU fixes the bug.
The remaining issue affects just one date, the “change-over” date when WU transitions to returning the correct data - and will only bite for stations east of UTC, and only in the hours that they are ahead of UTC. Currently, for me, that date is 2019-05-28. When it originally asks for 2019-05-28, it returns the data for 2019-05-29. Wunderfixer then tries to compensate by asking for 2019-05-27. But 2019-05-27 is a date before the WU bug, and so it returns the data for 2019-05-27. We can’t work around this, because no request to WU will actually return the data we want. The only fix is to wait a few hours until the UTC date is aligned with the Sydney date. Good news is that for stations west of UTC, this won’t happen as the bug causes a duplicate day, not a missing day. Here’s the output for the requests for my station from 2019-05-27 (before the WU bug kicks in) to today 2019-06-04. [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-05-27 --test --verbose | (head -n 11 ; tail -n 3) Using configuration file /home/weewx/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-05-27 WU API obsTimeLocal: 2019-05-27 epoch: 1558879200 date_epoch_local: 2019-05-27 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:00:00Z obsTimeLocal: 2019-05-27 00:00:00 epoch: 1558879500 date_epoch_local: 2019-05-27 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:05:00Z obsTimeLocal: 2019-05-27 00:05:00 epoch: 1558879800 date_epoch_local: 2019-05-27 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:10:00Z obsTimeLocal: 2019-05-27 00:10:00 epoch: 1558880100 date_epoch_local: 2019-05-27 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:15:00Z obsTimeLocal: 2019-05-27 00:15:00 epoch: 1558880400 date_epoch_local: 2019-05-27 00:20:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:20:00Z obsTimeLocal: 2019-05-27 00:20:00 epoch: 1558880700 date_epoch_local: 2019-05-27 00:25:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:25:00Z obsTimeLocal: 2019-05-27 00:25:00 epoch: 1558881000 date_epoch_local: 2019-05-27 00:30:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:30:00Z obsTimeLocal: 2019-05-27 00:30:00 epoch: 1558965000 date_epoch_local: 2019-05-27 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T13:50:00Z obsTimeLocal: 2019-05-27 23:50:00 epoch: 1558965300 date_epoch_local: 2019-05-27 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T13:55:00Z obsTimeLocal: 2019-05-27 23:55:00 Number of WU records: 288 [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-05-28 --test --verbose | (head -n 11 ; tail -n 3) No results returned from Weather Underground (perhaps a bad station name??). Using configuration file /home/weewx/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-05-28 WU API obsTimeLocal: 2019-05-29 WU API RETURNED WRONG DATE!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-05-27 WU API obsTimeLocal: 2019-05-27 WU API COMPENSATION FAILURE! ABORTING!!! Number of WU records: 0 [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-05-29 --test --verbose | (head -n 11 ; tail -n 3) Using configuration file /home/weewx/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-05-29 WU API obsTimeLocal: 2019-05-30 WU API RETURNED WRONG DATE!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-05-28 WU API obsTimeLocal: 2019-05-29 epoch: 1559052000 date_epoch_local: 2019-05-29 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T14:00:00Z obsTimeLocal: 2019-05-29 00:00:00 epoch: 1559052300 date_epoch_local: 2019-05-29 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T14:05:00Z obsTimeLocal: 2019-05-29 00:05:00 epoch: 1559052600 date_epoch_local: 2019-05-29 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T14:10:00Z obsTimeLocal: 2019-05-29 00:10:00 epoch: 1559052900 date_epoch_local: 2019-05-29 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T14:15:00Z obsTimeLocal: 2019-05-29 00:15:00 epoch: 1559137800 date_epoch_local: 2019-05-29 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T13:50:00Z obsTimeLocal: 2019-05-29 23:50:00 epoch: 1559138100 date_epoch_local: 2019-05-29 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T13:55:00Z obsTimeLocal: 2019-05-29 23:55:00 Number of WU records: 288 [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-05-30 --test --verbose | (head -n 11 ; tail -n 3) Using configuration file /home/weewx/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-05-30 WU API obsTimeLocal: 2019-05-31 WU API RETURNED WRONG DATE!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-05-29 WU API obsTimeLocal: 2019-05-30 epoch: 1559138400 date_epoch_local: 2019-05-30 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T14:00:00Z obsTimeLocal: 2019-05-30 00:00:00 epoch: 1559138700 date_epoch_local: 2019-05-30 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T14:05:00Z obsTimeLocal: 2019-05-30 00:05:00 epoch: 1559139000 date_epoch_local: 2019-05-30 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T14:10:00Z obsTimeLocal: 2019-05-30 00:10:00 epoch: 1559139300 date_epoch_local: 2019-05-30 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T14:15:00Z obsTimeLocal: 2019-05-30 00:15:00 epoch: 1559224200 date_epoch_local: 2019-05-30 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T13:50:00Z obsTimeLocal: 2019-05-30 23:50:00 epoch: 1559224500 date_epoch_local: 2019-05-30 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T13:55:00Z obsTimeLocal: 2019-05-30 23:55:00 Number of WU records: 288 [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-05-31 --test --verbose | (head -n 11 ; tail -n 3) Using configuration file /home/weewx/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-05-31 WU API obsTimeLocal: 2019-06-01 WU API RETURNED WRONG DATE!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-05-30 WU API obsTimeLocal: 2019-05-31 epoch: 1559224800 date_epoch_local: 2019-05-31 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:00:00Z obsTimeLocal: 2019-05-31 00:00:00 epoch: 1559225100 date_epoch_local: 2019-05-31 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:05:00Z obsTimeLocal: 2019-05-31 00:05:00 epoch: 1559225400 date_epoch_local: 2019-05-31 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:10:00Z obsTimeLocal: 2019-05-31 00:10:00 epoch: 1559225700 date_epoch_local: 2019-05-31 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:15:00Z obsTimeLocal: 2019-05-31 00:15:00 epoch: 1559310600 date_epoch_local: 2019-05-31 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T13:50:00Z obsTimeLocal: 2019-05-31 23:50:00 epoch: 1559310900 date_epoch_local: 2019-05-31 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T13:55:00Z obsTimeLocal: 2019-05-31 23:55:00 Number of WU records: 288 [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-06-01 --test --verbose | (head -n 11 ; tail -n 3) Using configuration file /home/weewx/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-06-01 WU API obsTimeLocal: 2019-06-02 WU API RETURNED WRONG DATE!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-05-31 WU API obsTimeLocal: 2019-06-01 epoch: 1559311200 date_epoch_local: 2019-06-01 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T14:00:00Z obsTimeLocal: 2019-06-01 00:00:00 epoch: 1559311500 date_epoch_local: 2019-06-01 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T14:05:00Z obsTimeLocal: 2019-06-01 00:05:00 epoch: 1559311800 date_epoch_local: 2019-06-01 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T14:10:00Z obsTimeLocal: 2019-06-01 00:10:00 epoch: 1559312100 date_epoch_local: 2019-06-01 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T14:15:00Z obsTimeLocal: 2019-06-01 00:15:00 epoch: 1559397000 date_epoch_local: 2019-06-01 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T13:50:00Z obsTimeLocal: 2019-06-01 23:50:00 epoch: 1559397300 date_epoch_local: 2019-06-01 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T13:55:00Z obsTimeLocal: 2019-06-01 23:55:00 Number of WU records: 288 [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-06-02 --test --verbose | (head -n 11 ; tail -n 3) Using configuration file /home/weewx/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-06-02 WU API obsTimeLocal: 2019-06-03 WU API RETURNED WRONG DATE!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-06-01 WU API obsTimeLocal: 2019-06-02 epoch: 1559397600 date_epoch_local: 2019-06-02 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 00:00:00 epoch: 1559397900 date_epoch_local: 2019-06-02 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 00:05:00 epoch: 1559398200 date_epoch_local: 2019-06-02 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:10:00Z obsTimeLocal: 2019-06-02 00:10:00 epoch: 1559398500 date_epoch_local: 2019-06-02 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:15:00Z obsTimeLocal: 2019-06-02 00:15:00 epoch: 1559483400 date_epoch_local: 2019-06-02 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T13:50:00Z obsTimeLocal: 2019-06-02 23:50:00 epoch: 1559483700 date_epoch_local: 2019-06-02 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T13:55:00Z obsTimeLocal: 2019-06-02 23:55:00 Number of WU records: 288 [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-06-03 --test --verbose | (head -n 11 ; tail -n 3) Using configuration file /home/weewx/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-06-03 WU API obsTimeLocal: 2019-06-04 WU API RETURNED WRONG DATE!!!!!!!!!!!!!!! WU API COMPENSATION DATE: 2019-06-02 WU API obsTimeLocal: 2019-06-03 epoch: 1559484000 date_epoch_local: 2019-06-03 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:00:00Z obsTimeLocal: 2019-06-03 00:00:00 epoch: 1559484300 date_epoch_local: 2019-06-03 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:05:00Z obsTimeLocal: 2019-06-03 00:05:00 epoch: 1559484600 date_epoch_local: 2019-06-03 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:10:00Z obsTimeLocal: 2019-06-03 00:10:00 epoch: 1559484900 date_epoch_local: 2019-06-03 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:15:00Z obsTimeLocal: 2019-06-03 00:15:00 epoch: 1559569800 date_epoch_local: 2019-06-03 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T13:50:00Z obsTimeLocal: 2019-06-03 23:50:00 epoch: 1559570100 date_epoch_local: 2019-06-03 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T13:55:00Z obsTimeLocal: 2019-06-03 23:55:00 Number of WU records: 288 [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --test --verbose | (head -n 11 ; tail -n 3) Using configuration file /home/weewx/weewx.conf. Weather Underground Station: ISYDNEY155 Date to check: 2019-06-04 WU API obsTimeLocal: 2019-06-04 epoch: 1559570400 date_epoch_local: 2019-06-04 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:00:00Z obsTimeLocal: 2019-06-04 00:00:00 epoch: 1559570700 date_epoch_local: 2019-06-04 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:05:00Z obsTimeLocal: 2019-06-04 00:05:00 epoch: 1559571000 date_epoch_local: 2019-06-04 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:10:00Z obsTimeLocal: 2019-06-04 00:10:00 epoch: 1559571300 date_epoch_local: 2019-06-04 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:15:00Z obsTimeLocal: 2019-06-04 00:15:00 epoch: 1559571600 date_epoch_local: 2019-06-04 00:20:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:20:00Z obsTimeLocal: 2019-06-04 00:20:00 epoch: 1559571900 date_epoch_local: 2019-06-04 00:25:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:25:00Z obsTimeLocal: 2019-06-04 00:25:00 epoch: 1559572200 date_epoch_local: 2019-06-04 00:30:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:30:00Z obsTimeLocal: 2019-06-04 00:30:00 epoch: 1559595300 date_epoch_local: 2019-06-04 06:55:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T20:55:00Z obsTimeLocal: 2019-06-04 06:55:00 epoch: 1559595600 date_epoch_local: 2019-06-04 07:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T21:00:00Z obsTimeLocal: 2019-06-04 07:00:00 Number of WU records: 85 Regards, Rod On 4 Jun 2019, at 5:34 am, Leon Shaner <l...@isylum.org<mailto:l...@isylum.org>> wrote: All, There are updates to the wunderdates utilities, to implement the following: Add +/- 1-day compensation logic to workaround WU 'historical' API bug, depending on whether station is east/west of UTC, and whether it is within X hours of UTC midnight, relative to the local station time. These wunderdates utilities are for debug purposes only to aid in testing the WU behavior, while IBM works to fix the actual bug (no intention of adding this logic to wunderfixer, which will remain broken until IBM fixes their API). The bugs are well known / well documented. The continued use of wunderdates is just to keep an eye on IBM's progress in fixing the bugs, e.g. keep an eye out for if the behavior changes / improves. Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) On Jun 1, 2019, at 1:17 PM, Leon Shaner <l...@isylum.org<mailto:l...@isylum.org>> wrote: All, I'm testing a new approach. Below you will find links to an updated wunderdates utility that can be used to validate whether I am on the right track. The wunderdates utility simply dumps out timestamp related records from what WU returns from the query. We're looking mainly at the obsTimeUtc and obsTimeLocal values, which were demonstrated under the prior approach to be returning the wrong dates when within the stations localtime +/- offset from UTC. The new approach is to detect if the requested date is 'today' and if so, use a different API URL that already assumes 'today' and will hopefully not be subject to the UTC offset bugs we've been chasing with the historical data API URL. I have my crontab set to do another test approaching my UTC offset, just after coming within the offset, and then again just before and after midnight localtime. (Same test I did before, but now with the new approach in place). Here is the utility for anyone else that wants to check out the behavior: https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3 https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4 Which version you pull will depend on which base weewx you are running. Pull the one that matches your weewx version and place it in bin, next to wunderfixer, and it will take the same arguments as wunderfixer. You can just try wunderfixer as you normally would (with --apikey) and then run wunderdates(3 or 4) with exact same arguments to be able to see what actual timestamps WU is sending back for the date queried. Parameters like --epsilon don't have any effect in the case of wunderdates, but I left it there so you don't have to change options when running the util to get the debug output. Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) On May 30, 2019, at 12:40 AM, Leon Shaner <l...@isylum.org<mailto:l...@isylum.org>> wrote: Rod, Thanks again for this. Since the in-progress version of wunderfixer doesn't really show you the dates that come back from WU, I have written a tool just to debug the dates. The command-line input and basing of defaults on weewx.conf works the same as wunderfixer, except it doesn't look at your DB at all. It only prints out datestamps in various incarnations coming back from WU and as compared to your system's localtime. It would be helpful to see the "wunderdates" output at times like you've shown below, a la before and after your localtime rolls around to UTC midnight. Since you are at UTC + 10, another interesting time would be 11+ hours on either side of UTC midnight, in addition to within 9 or less hours. This is just to make sure we're covering all the corner cases. Gary reported a difference for stations that are east vs. west of GMT, and I expect we're really chasing the same bug in that there is some bad math WU is doing based on UTC offset, but since an offset can be +/-, the effects go in opposite directions date-wise, depending on which side of the UTC dateline your station is located. At least that is what I surmise may be happening, but the wunderdates utility should shed light one way or the other. =D The wunderdates utility is available at the links below. https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3 https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4 Which version you pull will depend on which base weewx you are running. Pull the one that matches your weewx version and place it in bin, next to wunderfixer, and it will take the same arguments as wunderfixer. You can just try wunderfixer as you normally would (with --apikey) and then run wunderdates(3 or 4) with exact same arguments to be able to see what actual timestamps WU is sending back for the date queried. Parameters like --epsilon don't have any effect in the case of wunderdates, but I left it there so you don't have to change options when running the util to get the debug output. What I've been doing is saving the output to files for use with sdiff (side-by-side) diff. Or you can just compare the head and tail of each file individually. Example for my system before and after my local time on 5-28 once it was at/after 8 p.m. here, which is within 4 hours of UTC midnight (I am at UTC-4). This output is optimized for screen widths 203 or wider. Sorry. :S Mainly the last two data fields in the output tell what we need to know. pi@nixie:/var/tmp $ head -n 3 59:19:28:5_wu.txt 0:20:28:5_wu.txt ==> 59:19:28:5_wu.txt <== Using configuration file /usr/share/weewx4/weewx.conf. epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: 1559016699 date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59 epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: 1559016999 date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:09:59Z obsTimeLocal: 2019-05-28 00:09:59 ==> 0:20:28:5_wu.txt <== Using configuration file /usr/share/weewx4/weewx.conf. epoch: 1558929899 date_epoch_local: 2019-05-27 00:04:59 utcepoch: 1558930299 date_epoch_utc: 2019-05-27 04:04:59 tz: America/New_York obsTimeUtc: 2019-05-27T04:04:59Z obsTimeLocal: 2019-05-27 00:04:59 epoch: 1558930199 date_epoch_local: 2019-05-27 00:09:59 utcepoch: 1558930599 date_epoch_utc: 2019-05-27 04:09:59 tz: America/New_York obsTimeUtc: 2019-05-27T04:09:59Z obsTimeLocal: 2019-05-27 00:09:59 pi@nixie:/var/tmp $ tail -n 3 59:19:28:5_wu.txt 0:20:28:5_wu.txt ==> 59:19:28:5_wu.txt <== epoch: 1559087699 date_epoch_local: 2019-05-28 19:54:59 utcepoch: 1559088099 date_epoch_utc: 2019-05-28 23:54:59 tz: America/New_York obsTimeUtc: 2019-05-28T23:54:59Z obsTimeLocal: 2019-05-28 19:54:59 epoch: 1559087702 date_epoch_local: 2019-05-28 19:55:02 utcepoch: 1559088102 date_epoch_utc: 2019-05-28 23:55:02 tz: America/New_York obsTimeUtc: 2019-05-28T23:55:02Z obsTimeLocal: 2019-05-28 19:55:02 Number of WU records: 240 ==> 0:20:28:5_wu.txt <== epoch: 1559015699 date_epoch_local: 2019-05-27 23:54:59 utcepoch: 1559016099 date_epoch_utc: 2019-05-28 03:54:59 tz: America/New_York obsTimeUtc: 2019-05-28T03:54:59Z obsTimeLocal: 2019-05-27 23:54:59 epoch: 1559015999 date_epoch_local: 2019-05-27 23:59:59 utcepoch: 1559016399 date_epoch_utc: 2019-05-28 03:59:59 tz: America/New_York obsTimeUtc: 2019-05-28T03:59:59Z obsTimeLocal: 2019-05-27 23:59:59 Number of WU records: 288 But after my system rolled over midnight localtime, results returned to the correct dates when asking for 5-28: pi@nixie:/var/tmp $ head -3 0:0:29:5_wu.txt Using configuration file /usr/share/weewx4/weewx.conf. epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: 1559016699 date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59 epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: 1559016999 date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:09:59Z obsTimeLocal: 2019-05-28 00:09:59 pi@nixie:/var/tmp $ tail -3 0:0:29:5_wu.txt epoch: 1559102099 date_epoch_local: 2019-05-28 23:54:59 utcepoch: 1559102499 date_epoch_utc: 2019-05-29 03:54:59 tz: America/New_York obsTimeUtc: 2019-05-29T03:54:59Z obsTimeLocal: 2019-05-28 23:54:59 epoch: 1559102399 date_epoch_local: 2019-05-28 23:59:59 utcepoch: 1559102799 date_epoch_utc: 2019-05-29 03:59:59 tz: America/New_York obsTimeUtc: 2019-05-29T03:59:59Z obsTimeLocal: 2019-05-28 23:59:59 Number of WU records: 288 Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) On May 29, 2019, at 8:53 PM, Rod Yager <r...@yager.net.au<mailto:r...@yager.net.au>> wrote: Further to this, it has now rolled past 10am here, so the local date is now the same as the UTC date. (ie. local time May 30 2019 10:40 AM = May 30 2019 00:40 AM UTC). Now I get: ./wunderfixer --verbose --date=2019-05-29 --epsilon=125 Using configuration file /home/weewx/weewx.conf. Using database binding 'wx_binding', which is bound to database 'archive_mysql' Weather Underground Station: xxxxx Date to check: 2019-05-29 Number of archive records: 288 Number of WU records: 288 Number of missing records: 0 [root@moses bin]# ./wunderfixer --verbose --date=2019-05-30 --epsilon=125 Using configuration file /home/weewx/weewx.conf. Using database binding 'wx_binding', which is bound to database 'archive_mysql' Weather Underground Station: xxxxxx Date to check: 2019-05-30 Number of archive records: 128 Number of WU records: 127 Number of missing records: 1 This means that WU is now actually providing the records for the date requested, rather than the day after the requested date. So it seems that what is happening is that WU is determining whether or not the current date at the station location is the same as the UTC date. If it is, it returns the data for the date as in the request. But if the local date is different, it makes an (unwanted) adjustment for the date difference. Rod On Thursday, May 30, 2019 at 9:29:16 AM UTC+10, Leon Shaner wrote: Hi, Rod, Yes and thanks for adding yet another confirmation of the issue. =D I can show that if I do the query within X hours of my offset of UTC, what actually happens is they report 288 records from the day PRIOR to the one I am asking about. For example, I ask for 20190528 and they give me records for 20190527, so *that* is why wunderfixer "thinks" it needs to re-upload everything. I am in contact with IBM about it and have shown them irrefutable proof of the issue. They didn't respond back yet, which I expect is because the proof was irrefutable. Ha! ;-) I expect that they're investigating and would rather respond from a position of understanding, or with any luck maybe even a quick fix. =D I meant to follow-up with IBM again this morning, but got waylaid, so I'll do that now. Thanks again, and for the reminder. =D Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) On May 29, 2019, at 6:14 PM, Rod Yager <r...@yager.net.au> wrote: There is definitely a time zone issue. I am in the Sydney Australia timezone (UTC +10 hours). It is currently 8am local time on May 30, 2019. (10pm May 29, 2019 UTC) If I execute ./wunderfixer --verbose --date=2019-05-29 --epsilon=125 I get Using configuration file /home/weewx/weewx.conf. Using database binding 'wx_binding', which is bound to database 'archive_mysql' Weather Underground Station: xxxxxxx Date to check: 2019-05-29 Number of archive records: 288 Number of WU records: 97 Number of missing records: 288 Now WU actually has 288 records for 2019-05-29. But it only has 97 records for 2019-05-30. So it is clear that wunderfixer is downloading the record data for 2019-05-30 from WeatherUnderground and trying to match them with the local records for 2019-30-29. Of course, they all mismatch, and so wunderfixer concludes that it must upload all the data for 2019-05-29. Hope this narrows down the search for a solution. Rod On Monday, May 27, 2019 at 9:35:25 PM UTC+10, Leon Shaner wrote: On May 27, 2019, at 12:12 AM, gjr80 <gjrod...@gmail.com> wrote: On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote: [snip] If you can see any shorter paths to a more reliable outcome than I have achieved so far, then you know know know I will be very grateful. =D I am not sure what local/UTC issue you refer to. When I do a api.weather.com/v2/pws/history<http://api.weather.com/v2/pws/history> query on a station to the east of Greenwich I am returned all records for the date specified (eg 20190525 gives me all records for 25 May 2019), each record contains an epoch timestamp which is correct and consistent with 25 May 2019. Everything is as I would expect. However, when I perform the same query on a station to the west of Greenwich I am returned records for the day before the date specified (ie 20190525 gives me all records for 24 May 2019 not 25 May 2019), again each record contains an epoch timestamp but the timestamp is for the previous day Ie 24 May 2019. I have checked a number of data records in the stations history table and WU is definitely returning the midnight to midnight data for the day before. I have confirmed this behaviour with a number of stations both east and west of Greenwich. I don't think there is a local/UTC time issue, I think WU is having some implementation issues and for stations west of Greenwich they are returning the wrong day of data. Thanks, Gary! This was all very helpful. In addition to what you've described across the east vs west of GMT, I get similar behavior if I am within X hours of my local UTC offset when querying my own station. Last night as soon as localtime rolled over midnight, the queries for the previous day were correct. --Leon Gary -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/683a28af-35e4-474e-95a0-f684b9926af0%40googlegroups.com<https://groups.google.com/d/msgid/weewx-user/683a28af-35e4-474e-95a0-f684b9926af0%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com<mailto:weewx-user+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3570f548-caf3-43e8-858d-1b4d8c87d84a%40googlegroups.com<https://groups.google.com/d/msgid/weewx-user/3570f548-caf3-43e8-858d-1b4d8c87d84a%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com<mailto:weewx-user+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/388426A6-12D0-45FC-AFF2-E6EB532B61DC%40isylum.org<https://groups.google.com/d/msgid/weewx-user/388426A6-12D0-45FC-AFF2-E6EB532B61DC%40isylum.org?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com<mailto:weewx-user+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3C441E3C-B1C9-4FC4-88BB-6285A9157F46%40isylum.org<https://groups.google.com/d/msgid/weewx-user/3C441E3C-B1C9-4FC4-88BB-6285A9157F46%40isylum.org?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/F16953D6-6EEB-486C-9560-0A5A9F9357E8%40yager.net.au. For more options, visit https://groups.google.com/d/optout.
