Dear Leon, Your approach is flawed. It won’t work for historical data.
If you run wunderdates —date=2019-01-01 at 9pm your time you will receive the data for 2018-12-31 (because your station is a day behind UTC at the time of the request, so it gives you the data for the previous day). On the other hand, if you run wunderdates —date=2019-01-01 at 9am your time you will receive the data for 2019-01-01 (because your station is in the same day as UTC at the time of the request). If I do the same thing (I’m in UTC+10) wunderdates —date=2019-01-01 at 9pm my time produces the data for 2019-01-01 (because my station is in the same day as UTC at the time of the request) but wunderdates —date=2019-01-01 at 9am my time produces the data for 2019-01-02 (because my station is a day ahead of UTC at the time of the request) Rod On 2 Jun 2019, at 3:17 am, Leon Shaner <l...@isylum.org<mailto:l...@isylum.org>> wrote: All, I'm testing a new approach. Below you will find links to an updated wunderdates utility that can be used to validate whether I am on the right track. The wunderdates utility simply dumps out timestamp related records from what WU returns from the query. We're looking mainly at the obsTimeUtc and obsTimeLocal values, which were demonstrated under the prior approach to be returning the wrong dates when within the stations localtime +/- offset from UTC. The new approach is to detect if the requested date is 'today' and if so, use a different API URL that already assumes 'today' and will hopefully not be subject to the UTC offset bugs we've been chasing with the historical data API URL. I have my crontab set to do another test approaching my UTC offset, just after coming within the offset, and then again just before and after midnight localtime. (Same test I did before, but now with the new approach in place). Here is the utility for anyone else that wants to check out the behavior: https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3 https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4 Which version you pull will depend on which base weewx you are running. Pull the one that matches your weewx version and place it in bin, next to wunderfixer, and it will take the same arguments as wunderfixer. You can just try wunderfixer as you normally would (with --apikey) and then run wunderdates(3 or 4) with exact same arguments to be able to see what actual timestamps WU is sending back for the date queried. Parameters like --epsilon don't have any effect in the case of wunderdates, but I left it there so you don't have to change options when running the util to get the debug output. Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) On May 30, 2019, at 12:40 AM, Leon Shaner <l...@isylum.org<mailto:l...@isylum.org>> wrote: Rod, Thanks again for this. Since the in-progress version of wunderfixer doesn't really show you the dates that come back from WU, I have written a tool just to debug the dates. The command-line input and basing of defaults on weewx.conf works the same as wunderfixer, except it doesn't look at your DB at all. It only prints out datestamps in various incarnations coming back from WU and as compared to your system's localtime. It would be helpful to see the "wunderdates" output at times like you've shown below, a la before and after your localtime rolls around to UTC midnight. Since you are at UTC + 10, another interesting time would be 11+ hours on either side of UTC midnight, in addition to within 9 or less hours. This is just to make sure we're covering all the corner cases. Gary reported a difference for stations that are east vs. west of GMT, and I expect we're really chasing the same bug in that there is some bad math WU is doing based on UTC offset, but since an offset can be +/-, the effects go in opposite directions date-wise, depending on which side of the UTC dateline your station is located. At least that is what I surmise may be happening, but the wunderdates utility should shed light one way or the other. =D The wunderdates utility is available at the links below. https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3 https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4 Which version you pull will depend on which base weewx you are running. Pull the one that matches your weewx version and place it in bin, next to wunderfixer, and it will take the same arguments as wunderfixer. You can just try wunderfixer as you normally would (with --apikey) and then run wunderdates(3 or 4) with exact same arguments to be able to see what actual timestamps WU is sending back for the date queried. Parameters like --epsilon don't have any effect in the case of wunderdates, but I left it there so you don't have to change options when running the util to get the debug output. What I've been doing is saving the output to files for use with sdiff (side-by-side) diff. Or you can just compare the head and tail of each file individually. Example for my system before and after my local time on 5-28 once it was at/after 8 p.m. here, which is within 4 hours of UTC midnight (I am at UTC-4). This output is optimized for screen widths 203 or wider. Sorry. :S Mainly the last two data fields in the output tell what we need to know. pi@nixie:/var/tmp $ head -n 3 59:19:28:5_wu.txt 0:20:28:5_wu.txt ==> 59:19:28:5_wu.txt <== Using configuration file /usr/share/weewx4/weewx.conf. epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: 1559016699 date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59 epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: 1559016999 date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:09:59Z obsTimeLocal: 2019-05-28 00:09:59 ==> 0:20:28:5_wu.txt <== Using configuration file /usr/share/weewx4/weewx.conf. epoch: 1558929899 date_epoch_local: 2019-05-27 00:04:59 utcepoch: 1558930299 date_epoch_utc: 2019-05-27 04:04:59 tz: America/New_York obsTimeUtc: 2019-05-27T04:04:59Z obsTimeLocal: 2019-05-27 00:04:59 epoch: 1558930199 date_epoch_local: 2019-05-27 00:09:59 utcepoch: 1558930599 date_epoch_utc: 2019-05-27 04:09:59 tz: America/New_York obsTimeUtc: 2019-05-27T04:09:59Z obsTimeLocal: 2019-05-27 00:09:59 pi@nixie:/var/tmp $ tail -n 3 59:19:28:5_wu.txt 0:20:28:5_wu.txt ==> 59:19:28:5_wu.txt <== epoch: 1559087699 date_epoch_local: 2019-05-28 19:54:59 utcepoch: 1559088099 date_epoch_utc: 2019-05-28 23:54:59 tz: America/New_York obsTimeUtc: 2019-05-28T23:54:59Z obsTimeLocal: 2019-05-28 19:54:59 epoch: 1559087702 date_epoch_local: 2019-05-28 19:55:02 utcepoch: 1559088102 date_epoch_utc: 2019-05-28 23:55:02 tz: America/New_York obsTimeUtc: 2019-05-28T23:55:02Z obsTimeLocal: 2019-05-28 19:55:02 Number of WU records: 240 ==> 0:20:28:5_wu.txt <== epoch: 1559015699 date_epoch_local: 2019-05-27 23:54:59 utcepoch: 1559016099 date_epoch_utc: 2019-05-28 03:54:59 tz: America/New_York obsTimeUtc: 2019-05-28T03:54:59Z obsTimeLocal: 2019-05-27 23:54:59 epoch: 1559015999 date_epoch_local: 2019-05-27 23:59:59 utcepoch: 1559016399 date_epoch_utc: 2019-05-28 03:59:59 tz: America/New_York obsTimeUtc: 2019-05-28T03:59:59Z obsTimeLocal: 2019-05-27 23:59:59 Number of WU records: 288 But after my system rolled over midnight localtime, results returned to the correct dates when asking for 5-28: pi@nixie:/var/tmp $ head -3 0:0:29:5_wu.txt Using configuration file /usr/share/weewx4/weewx.conf. epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: 1559016699 date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59 epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: 1559016999 date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:09:59Z obsTimeLocal: 2019-05-28 00:09:59 pi@nixie:/var/tmp $ tail -3 0:0:29:5_wu.txt epoch: 1559102099 date_epoch_local: 2019-05-28 23:54:59 utcepoch: 1559102499 date_epoch_utc: 2019-05-29 03:54:59 tz: America/New_York obsTimeUtc: 2019-05-29T03:54:59Z obsTimeLocal: 2019-05-28 23:54:59 epoch: 1559102399 date_epoch_local: 2019-05-28 23:59:59 utcepoch: 1559102799 date_epoch_utc: 2019-05-29 03:59:59 tz: America/New_York obsTimeUtc: 2019-05-29T03:59:59Z obsTimeLocal: 2019-05-28 23:59:59 Number of WU records: 288 Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) On May 29, 2019, at 8:53 PM, Rod Yager <r...@yager.net.au<mailto:r...@yager.net.au>> wrote: Further to this, it has now rolled past 10am here, so the local date is now the same as the UTC date. (ie. local time May 30 2019 10:40 AM = May 30 2019 00:40 AM UTC). Now I get: ./wunderfixer --verbose --date=2019-05-29 --epsilon=125 Using configuration file /home/weewx/weewx.conf. Using database binding 'wx_binding', which is bound to database 'archive_mysql' Weather Underground Station: xxxxx Date to check: 2019-05-29 Number of archive records: 288 Number of WU records: 288 Number of missing records: 0 [root@moses bin]# ./wunderfixer --verbose --date=2019-05-30 --epsilon=125 Using configuration file /home/weewx/weewx.conf. Using database binding 'wx_binding', which is bound to database 'archive_mysql' Weather Underground Station: xxxxxx Date to check: 2019-05-30 Number of archive records: 128 Number of WU records: 127 Number of missing records: 1 This means that WU is now actually providing the records for the date requested, rather than the day after the requested date. So it seems that what is happening is that WU is determining whether or not the current date at the station location is the same as the UTC date. If it is, it returns the data for the date as in the request. But if the local date is different, it makes an (unwanted) adjustment for the date difference. Rod On Thursday, May 30, 2019 at 9:29:16 AM UTC+10, Leon Shaner wrote: Hi, Rod, Yes and thanks for adding yet another confirmation of the issue. =D I can show that if I do the query within X hours of my offset of UTC, what actually happens is they report 288 records from the day PRIOR to the one I am asking about. For example, I ask for 20190528 and they give me records for 20190527, so *that* is why wunderfixer "thinks" it needs to re-upload everything. I am in contact with IBM about it and have shown them irrefutable proof of the issue. They didn't respond back yet, which I expect is because the proof was irrefutable. Ha! ;-) I expect that they're investigating and would rather respond from a position of understanding, or with any luck maybe even a quick fix. =D I meant to follow-up with IBM again this morning, but got waylaid, so I'll do that now. Thanks again, and for the reminder. =D Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) On May 29, 2019, at 6:14 PM, Rod Yager <r...@yager.net.au> wrote: There is definitely a time zone issue. I am in the Sydney Australia timezone (UTC +10 hours). It is currently 8am local time on May 30, 2019. (10pm May 29, 2019 UTC) If I execute ./wunderfixer --verbose --date=2019-05-29 --epsilon=125 I get Using configuration file /home/weewx/weewx.conf. Using database binding 'wx_binding', which is bound to database 'archive_mysql' Weather Underground Station: xxxxxxx Date to check: 2019-05-29 Number of archive records: 288 Number of WU records: 97 Number of missing records: 288 Now WU actually has 288 records for 2019-05-29. But it only has 97 records for 2019-05-30. So it is clear that wunderfixer is downloading the record data for 2019-05-30 from WeatherUnderground and trying to match them with the local records for 2019-30-29. Of course, they all mismatch, and so wunderfixer concludes that it must upload all the data for 2019-05-29. Hope this narrows down the search for a solution. Rod On Monday, May 27, 2019 at 9:35:25 PM UTC+10, Leon Shaner wrote: On May 27, 2019, at 12:12 AM, gjr80 <gjrod...@gmail.com> wrote: On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote: [snip] If you can see any shorter paths to a more reliable outcome than I have achieved so far, then you know know know I will be very grateful. =D I am not sure what local/UTC issue you refer to. When I do a api.weather.com/v2/pws/history<http://api.weather.com/v2/pws/history> query on a station to the east of Greenwich I am returned all records for the date specified (eg 20190525 gives me all records for 25 May 2019), each record contains an epoch timestamp which is correct and consistent with 25 May 2019. Everything is as I would expect. However, when I perform the same query on a station to the west of Greenwich I am returned records for the day before the date specified (ie 20190525 gives me all records for 24 May 2019 not 25 May 2019), again each record contains an epoch timestamp but the timestamp is for the previous day Ie 24 May 2019. I have checked a number of data records in the stations history table and WU is definitely returning the midnight to midnight data for the day before. I have confirmed this behaviour with a number of stations both east and west of Greenwich. I don't think there is a local/UTC time issue, I think WU is having some implementation issues and for stations west of Greenwich they are returning the wrong day of data. Thanks, Gary! This was all very helpful. In addition to what you've described across the east vs west of GMT, I get similar behavior if I am within X hours of my local UTC offset when querying my own station. Last night as soon as localtime rolled over midnight, the queries for the previous day were correct. --Leon Gary -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/683a28af-35e4-474e-95a0-f684b9926af0%40googlegroups.com<https://groups.google.com/d/msgid/weewx-user/683a28af-35e4-474e-95a0-f684b9926af0%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com<mailto:weewx-user+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3570f548-caf3-43e8-858d-1b4d8c87d84a%40googlegroups.com<https://groups.google.com/d/msgid/weewx-user/3570f548-caf3-43e8-858d-1b4d8c87d84a%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com<mailto:weewx-user+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/388426A6-12D0-45FC-AFF2-E6EB532B61DC%40isylum.org<https://groups.google.com/d/msgid/weewx-user/388426A6-12D0-45FC-AFF2-E6EB532B61DC%40isylum.org?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com<mailto:weewx-user+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/184A035F-2BFD-4C43-9584-B84EB083F11F%40isylum.org<https://groups.google.com/d/msgid/weewx-user/184A035F-2BFD-4C43-9584-B84EB083F11F%40isylum.org?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/D30A437D-57EF-46B0-A40F-98A7C1D2D793%40yager.net.au. For more options, visit https://groups.google.com/d/optout.
