Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Ray Mansell]

Following an extended local internet outage yesterday, I found myself with 
several hours' worth of data missing from WU. I recalled wunderfix, tried 
it, encountered the 403 problem, and eventually discovered this thread. 
Having read most of it, I tried looking for the modified 3.9.1 version on 
Github, but it seems not to be there, unless I'm looking in the wrong place.

So, I wonder if someone would be so kind as to answer the following?

1. Is the 3.9.1 apikey-modified wunderfix still relevant? I.e. does it work?
2. If so, where might I find it?

Thank you.

Ray

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/94e82df1-97bd-4577-9f6a-3262f48fa6ba%40googlegroups.com.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Thomas Keffer]

I didn't touch how wunderfixer posts (uploads) the fixes, only how it 
downloads data.

-tk



On Thursday, July 18, 2019 at 12:21:58 PM UTC-7, Leon Shaner wrote:
>
> Tom,
>
> There's nothing wrong with the old methods of reposting old records to WU.
>
> IBM is pretty clueless about the old interfaces we use from the pre-IBM WU 
> days.  When IBM says they don't yet support re-uploading old records via 
> API, they are most surely talking about the new API's.
>
> I can attest the methods that wunderfixer uses to re-upload do still work, 
> but they are subject to the old caveats that I mentioned in a thread some 
> while back.  WU quite clearly "normalizes" records to 5-minute boundaries 
> even if weewx is using rapidfire or an upload interval shorter than 
> 5-minutes.
>
> SO you can keep trying to re-upload records at n-minute or n-second 
> boundaries < 5-minutes, but they won't "stick."
>  Furthermore, even re-uploading of records that are aligned to 5-minute 
> boundaries seemed to take several re-uploads before they stick, but in 
> actuality it's just that they can take a very long time to "post" 
> full-circle such that they become available on a subsequent query (that was 
> the case even with the old query interface, and is still the case with the 
> new API for querying "today" and the other API for querying past days).
>
>  I have found that if I just spread out the calls to wunderfixer enough, 
> then the re-uploads do stick (after waiting long enough) and I don't need 
> to waste cycles re-uploading a second, third, or n-th time.
>
> Below is a recent example after a USB hang issue on my RPI.  You can see 
> how frequently my watchdog script runs, and what it found each time and 
> that in fact it was successful in re-uploading the missing records.
> The only exception seems to be the records very close to midnight tend to 
> usually not stick because WU is strangely keeping a record at 11:59:59.59 
> instead of accepting the 00:00:00.00  record.
>
> NOTE:  That in my case I have altered the weewx db query to pre-align the 
> local records to 5-minute boundaries (even though my archive interval is 
> every 1-minute). Pre-filtering the weewx archive query to 5-minute 
> boundaries cuts out a huge number of "false-positive" missing records that 
> wouldn't "stick" on the WU side even if I did let wunderfixer try to upload 
> them.   NOTE2:  Those every 1-minute records didn't "stick" even before IBM 
> went about breaking things wholesale across the board.
>
> Example (wunderfixer with --espsilon 125 --timeout 30, and with archive 
> queries normalized to 5-minute boundaries):
>
> Wed 17 Jul 01:17:01 EDT 2019 WeeWX: Wunderfixer timestamp: (1563340621)
> Wed 17 Jul 01:17:01 EDT 2019 Using configuration file 
> /usr/share/weewx4/weewx.conf.
> Wed 17 Jul 01:17:01 EDT 2019 Using database binding 'wx_binding', which is 
> bound to database 'archive_sqlite'
> Wed 17 Jul 01:17:01 EDT 2019 Weather Underground Station:   KMIDEARB5
> Wed 17 Jul 01:17:01 EDT 2019 Date to check: 2019-07-16
> Wed 17 Jul 01:17:01 EDT 2019 Number of archive records: 275
> Wed 17 Jul 01:17:01 EDT 2019 Number of WU records:  273
> Wed 17 Jul 01:17:01 EDT 2019 Number of missing records: 3
> Wed 17 Jul 01:17:01 EDT 2019
> Wed 17 Jul 01:17:01 EDT 2019 Missing records:
> Wed 17 Jul 01:17:01 EDT 2019 2019-07-16 00:00:00 EDT (1563249600); 
> 29.279";  73.2F;  97%; 0.0 mph; N/A deg; 0.0 mph gust;  72.3F; 0.00" rain 
>  ...published.
> Wed 17 Jul 01:17:01 EDT 2019 2019-07-16 23:55:00 EDT (1563335700); 
> 29.167";  72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain 
>  ...published.
> Wed 17 Jul 01:17:01 EDT 2019 2019-07-16 23:59:00 EDT (1563335940); 
> 29.167";  72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain 
>  ...published.
> Wed 17 Jul 01:17:01 EDT 2019 Using configuration file 
> /usr/share/weewx4/weewx.conf.
> Wed 17 Jul 01:17:01 EDT 2019 Using database binding 'wx_binding', which is 
> bound to database 'archive_sqlite'
> Wed 17 Jul 01:17:01 EDT 2019 Weather Underground Station:   KMIDEARB5
> Wed 17 Jul 01:17:01 EDT 2019 Date to check: 2019-07-17
> Wed 17 Jul 01:17:01 EDT 2019 Number of archive records: 17
> Wed 17 Jul 01:17:01 EDT 2019 Number of WU records:  15
> Wed 17 Jul 01:17:01 EDT 2019 Number of missing records: 3
> Wed 17 Jul 01:17:01 EDT 2019
> Wed 17 Jul 01:17:01 EDT 2019 Missing records:
> Wed 17 Jul 01:17:01 EDT 2019 2019-07-17 00:00:00 EDT (1563336000); 
> 29.167";  72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain 
>  ...published.
> Wed 17 Jul 01:17:01 EDT 2019 2019-07-17 01:15:00 EDT (1563340500); 
> 29.173";  72.5F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  72.2F; 0.00" rain 
>  ...published.
> Wed 17 Jul 01:17:01 EDT 2019 2019-07-17 01:16:00 EDT (1563340560); 
> 29.173";  72.5F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  72.2F; 0.00" rain 
>  ...published.
> Wed 17 Jul 01:22:03 EDT 2019 WeeWX: Wunderfixer has be

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Tom,

There's nothing wrong with the old methods of reposting old records to WU.

IBM is pretty clueless about the old interfaces we use from the pre-IBM WU 
days.  When IBM says they don't yet support re-uploading old records via API, 
they are most surely talking about the new API's.

I can attest the methods that wunderfixer uses to re-upload do still work, but 
they are subject to the old caveats that I mentioned in a thread some while 
back.  WU quite clearly "normalizes" records to 5-minute boundaries even if 
weewx is using rapidfire or an upload interval shorter than 5-minutes.

SO you can keep trying to re-upload records at n-minute or n-second boundaries 
< 5-minutes, but they won't "stick."
 Furthermore, even re-uploading of records that are aligned to 5-minute 
boundaries seemed to take several re-uploads before they stick, but in 
actuality it's just that they can take a very long time to "post" full-circle 
such that they become available on a subsequent query (that was the case even 
with the old query interface, and is still the case with the new API for 
querying "today" and the other API for querying past days).

 I have found that if I just spread out the calls to wunderfixer enough, then 
the re-uploads do stick (after waiting long enough) and I don't need to waste 
cycles re-uploading a second, third, or n-th time.

Below is a recent example after a USB hang issue on my RPI.  You can see how 
frequently my watchdog script runs, and what it found each time and that in 
fact it was successful in re-uploading the missing records.
The only exception seems to be the records very close to midnight tend to 
usually not stick because WU is strangely keeping a record at 11:59:59.59 
instead of accepting the 00:00:00.00  record.

NOTE:  That in my case I have altered the weewx db query to pre-align the local 
records to 5-minute boundaries (even though my archive interval is every 
1-minute). Pre-filtering the weewx archive query to 5-minute boundaries cuts 
out a huge number of "false-positive" missing records that wouldn't "stick" on 
the WU side even if I did let wunderfixer try to upload them.   NOTE2:  Those 
every 1-minute records didn't "stick" even before IBM went about breaking 
things wholesale across the board.

Example (wunderfixer with --espsilon 125 --timeout 30, and with archive queries 
normalized to 5-minute boundaries):

Wed 17 Jul 01:17:01 EDT 2019 WeeWX: Wunderfixer timestamp: (1563340621)
Wed 17 Jul 01:17:01 EDT 2019 Using configuration file 
/usr/share/weewx4/weewx.conf.
Wed 17 Jul 01:17:01 EDT 2019 Using database binding 'wx_binding', which is 
bound to database 'archive_sqlite'
Wed 17 Jul 01:17:01 EDT 2019 Weather Underground Station:   KMIDEARB5
Wed 17 Jul 01:17:01 EDT 2019 Date to check: 2019-07-16
Wed 17 Jul 01:17:01 EDT 2019 Number of archive records: 275
Wed 17 Jul 01:17:01 EDT 2019 Number of WU records:  273
Wed 17 Jul 01:17:01 EDT 2019 Number of missing records: 3
Wed 17 Jul 01:17:01 EDT 2019
Wed 17 Jul 01:17:01 EDT 2019 Missing records:
Wed 17 Jul 01:17:01 EDT 2019 2019-07-16 00:00:00 EDT (1563249600); 29.279";  
73.2F;  97%; 0.0 mph; N/A deg; 0.0 mph gust;  72.3F; 0.00" rain  ...published.
Wed 17 Jul 01:17:01 EDT 2019 2019-07-16 23:55:00 EDT (1563335700); 29.167";  
72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain  ...published.
Wed 17 Jul 01:17:01 EDT 2019 2019-07-16 23:59:00 EDT (1563335940); 29.167";  
72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain  ...published.
Wed 17 Jul 01:17:01 EDT 2019 Using configuration file 
/usr/share/weewx4/weewx.conf.
Wed 17 Jul 01:17:01 EDT 2019 Using database binding 'wx_binding', which is 
bound to database 'archive_sqlite'
Wed 17 Jul 01:17:01 EDT 2019 Weather Underground Station:   KMIDEARB5
Wed 17 Jul 01:17:01 EDT 2019 Date to check: 2019-07-17
Wed 17 Jul 01:17:01 EDT 2019 Number of archive records: 17
Wed 17 Jul 01:17:01 EDT 2019 Number of WU records:  15
Wed 17 Jul 01:17:01 EDT 2019 Number of missing records: 3
Wed 17 Jul 01:17:01 EDT 2019
Wed 17 Jul 01:17:01 EDT 2019 Missing records:
Wed 17 Jul 01:17:01 EDT 2019 2019-07-17 00:00:00 EDT (1563336000); 29.167";  
72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain  ...published.
Wed 17 Jul 01:17:01 EDT 2019 2019-07-17 01:15:00 EDT (1563340500); 29.173";  
72.5F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  72.2F; 0.00" rain  ...published.
Wed 17 Jul 01:17:01 EDT 2019 2019-07-17 01:16:00 EDT (1563340560); 29.173";  
72.5F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  72.2F; 0.00" rain  ...published.
Wed 17 Jul 01:22:03 EDT 2019 WeeWX: Wunderfixer has been intentionally skipped 
this iteration.
Wed 17 Jul 01:32:04 EDT 2019 WeeWX: Wunderfixer has been intentionally skipped 
this iteration.
Wed 17 Jul 01:42:03 EDT 2019 WeeWX: Wunderfixer timestamp: (1563342124)
Wed 17 Jul 01:42:03 EDT 2019 Using configuration file 
/usr/share/weewx4/weewx.conf.
Wed 17 Jul 01:42:03 EDT 2019 Using databas

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Thomas Keffer]

I've ported wunderfixer to the new WU API. Commit 32c35ce 

 on 
the development branch.


Unfortunately, it looks like the WU no longer allows re-posting old 
records, so the utility may no longer be useful. I'd be interested in other 
people's experience.


Also, as I understand it, there are issues for people living east of 
Greenwich. I live west, so I wasn't able to check that. Other people's 
experiences would be welcome.


NB: this is on the *development* branch. You will have to clone the weewx 
GitHub repository, then check out the development branch. 


-tk

On Monday, June 3, 2019 at 3:21:54 PM UTC-7, Rod Yager wrote:
>
> I’ve already migrated the today logic into my local version of 
> wunderfixer, which runs as a cron job at 05:58, 11:58, 17:58 and 23:58. 
>
> This works as expected here.
>
> The only circumstance I can see in which it would break is if I make a 
> request for today’s data  at 23:59:59.95 on my machine. The transmission 
> delay (and possibly differences in the clocks) will mean that WU receives 
> my request at a time when it thinks it is 00:00:00.01 on the following day 
> (my time) and so it will return the (empty) data for the day after the date 
> on which I made the request. We can’t worry about such things.
>
>
> Rod
>
>
>
> On 4 Jun 2019, at 7:55 am, Leon Shaner  wrote:
>
> Thanks, Rod!  =D 
>
> Those are precisely the same tests I ran and exact same results that 
> noted, before  before publishing.  =D 
> Plus a ton of checks against my own station, KMIDEARB5, west of UTC.
>
> As before, even more important than these 'historical' API tests is to 
> query against the current day at different times throughout the day, such 
> as before and after your midnight and before and after UTC midnight.  That 
> logic which uses the 'today' API (rapid) for today is the one we will 
> likely keep no matter what IBM does with the well-known bugs in the 
> 'historical' API.
>
> (I can't test the 'today' logic 100% without changing my localtime on my 
> system and I don't want to do that, which is why I appreciate help from 
> others to validate the code).  =D
>
> Regards, 
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>
> On Jun 3, 2019, at 5:39 PM, Rod Yager  wrote:
>
> Dear Leon, 
>
> Thanks for all your work on this. This works as well as we are going to be 
> able to manage unless and until WU fixes the bug. 
>
> The remaining issue affects just one date, the  “change-over” date when WU 
> transitions to returning the correct data - and will only bite for stations 
> east of UTC, and only in the hours that they are ahead of UTC.
>
> Currently, for me, that date is 2019-05-28. When it originally asks for 
> 2019-05-28, it returns the data for 2019-05-29. Wunderfixer then tries to 
> compensate by asking for 2019-05-27. But 2019-05-27 is a date before the WU 
> bug, and so it returns the data for 2019-05-27. We can’t work around this, 
> because no request to WU will actually return the data we want. The only 
> fix is to wait a few hours until the UTC date is aligned with the Sydney 
> date.
>
> Good news is that for stations west of UTC, this won’t happen as the bug 
> causes a duplicate day, not a missing day.
>
> Here’s the output for the requests for my station from 2019-05-27 (before 
> the WU bug kicks in) to today 2019-06-04.
>
>
> [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
> 2019-05-27  --test --verbose | (head -n 11 ; tail -n 3)
> Using configuration file /home/weewx/weewx.conf.
> Weather Underground Station:   ISYDNEY155
> Date to check: 2019-05-27
> WU API obsTimeLocal:   2019-05-27
> epoch: 1558879200 date_epoch_local: 2019-05-27 00:00:00 tz: 
> Australia/Sydney obsTimeUtc: 2019-05-26T14:00:00Z obsTimeLocal: 2019-05-27 
> 00:00:00
> epoch: 1558879500 date_epoch_local: 2019-05-27 00:05:00 tz: 
> Australia/Sydney obsTimeUtc: 2019-05-26T14:05:00Z obsTimeLocal: 2019-05-27 
> 00:05:00
> epoch: 1558879800 date_epoch_local: 2019-05-27 00:10:00 tz: 
> Australia/Sydney obsTimeUtc: 2019-05-26T14:10:00Z obsTimeLocal: 2019-05-27 
> 00:10:00
> epoch: 1558880100 date_epoch_local: 2019-05-27 00:15:00 tz: 
> Australia/Sydney obsTimeUtc: 2019-05-26T14:15:00Z obsTimeLocal: 2019-05-27 
> 00:15:00
> epoch: 1558880400 date_epoch_local: 2019-05-27 00:20:00 tz: 
> Australia/Sydney obsTimeUtc: 2019-05-26T14:20:00Z obsTimeLocal: 2019-05-27 
> 00:20:00
> epoch: 1558880700 date_epoch_local: 2019-05-27 00:25:00 tz: 
> Australia/Sydney obsTimeUtc: 2019-05-26T14:25:00Z obsTimeLocal: 2019-05-27 
> 00:25:00
> epoch: 1558881000 date_epoch_local: 2019-05-27 00:30:00 tz: 
> Australia/Sydney obsTimeUtc: 2019-05-26T14:30:00Z obsTimeLocal: 2019-05-27 
> 00:30:00
> epoch: 1558965000 date_epoch_local: 2019-05-27 23:50:00 tz: 
> Australia/Sydney obsTimeUtc: 2019-05-27T13:50:00Z obsTimeLocal: 2019-05-27 
> 23:50:00
> epoch: 1558965300 date_epoch_local: 2019-05-27

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Rod Yager]

I’ve already migrated the today logic into my local version of wunderfixer, 
which runs as a cron job at 05:58, 11:58, 17:58 and 23:58.

This works as expected here.

The only circumstance I can see in which it would break is if I make a request 
for today’s data  at 23:59:59.95 on my machine. The transmission delay (and 
possibly differences in the clocks) will mean that WU receives my request at a 
time when it thinks it is 00:00:00.01 on the following day (my time) and so it 
will return the (empty) data for the day after the date on which I made the 
request. We can’t worry about such things.


Rod



On 4 Jun 2019, at 7:55 am, Leon Shaner 
mailto:l...@isylum.org>> wrote:

Thanks, Rod!  =D

Those are precisely the same tests I ran and exact same results that noted, 
before  before publishing.  =D
Plus a ton of checks against my own station, KMIDEARB5, west of UTC.

As before, even more important than these 'historical' API tests is to query 
against the current day at different times throughout the day, such as before 
and after your midnight and before and after UTC midnight.  That logic which 
uses the 'today' API (rapid) for today is the one we will likely keep no matter 
what IBM does with the well-known bugs in the 'historical' API.

(I can't test the 'today' logic 100% without changing my localtime on my system 
and I don't want to do that, which is why I appreciate help from others to 
validate the code).  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

On Jun 3, 2019, at 5:39 PM, Rod Yager 
mailto:r...@yager.net.au>> wrote:

Dear Leon,

Thanks for all your work on this. This works as well as we are going to be able 
to manage unless and until WU fixes the bug.

The remaining issue affects just one date, the  “change-over” date when WU 
transitions to returning the correct data - and will only bite for stations 
east of UTC, and only in the hours that they are ahead of UTC.

Currently, for me, that date is 2019-05-28. When it originally asks for 
2019-05-28, it returns the data for 2019-05-29. Wunderfixer then tries to 
compensate by asking for 2019-05-27. But 2019-05-27 is a date before the WU 
bug, and so it returns the data for 2019-05-27. We can’t work around this, 
because no request to WU will actually return the data we want. The only fix is 
to wait a few hours until the UTC date is aligned with the Sydney date.

Good news is that for stations west of UTC, this won’t happen as the bug causes 
a duplicate day, not a missing day.

Here’s the output for the requests for my station from 2019-05-27 (before the 
WU bug kicks in) to today 2019-06-04.


[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
2019-05-27  --test --verbose | (head -n 11 ; tail -n 3)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-05-27
WU API obsTimeLocal:   2019-05-27
epoch: 1558879200 date_epoch_local: 2019-05-27 00:00:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:00:00Z obsTimeLocal: 2019-05-27 00:00:00
epoch: 1558879500 date_epoch_local: 2019-05-27 00:05:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:05:00Z obsTimeLocal: 2019-05-27 00:05:00
epoch: 1558879800 date_epoch_local: 2019-05-27 00:10:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:10:00Z obsTimeLocal: 2019-05-27 00:10:00
epoch: 1558880100 date_epoch_local: 2019-05-27 00:15:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:15:00Z obsTimeLocal: 2019-05-27 00:15:00
epoch: 1558880400 date_epoch_local: 2019-05-27 00:20:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:20:00Z obsTimeLocal: 2019-05-27 00:20:00
epoch: 1558880700 date_epoch_local: 2019-05-27 00:25:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:25:00Z obsTimeLocal: 2019-05-27 00:25:00
epoch: 1558881000 date_epoch_local: 2019-05-27 00:30:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:30:00Z obsTimeLocal: 2019-05-27 00:30:00
epoch: 1558965000 date_epoch_local: 2019-05-27 23:50:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-27T13:50:00Z obsTimeLocal: 2019-05-27 23:50:00
epoch: 1558965300 date_epoch_local: 2019-05-27 23:55:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-27T13:55:00Z obsTimeLocal: 2019-05-27 23:55:00
Number of WU records:  288
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
2019-05-28  --test --verbose | (head -n 11 ; tail -n 3)

No results returned from Weather Underground (perhaps a bad station name??).
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-05-28
WU API obsTimeLocal:   2019-05-29
WU API RETURNED WRONG DATE!!!
WU API COMPENSATION DATE:  2019-05-27
WU API obsTimeLocal:   2019-05-27
WU API COMPENSATION FAILURE! ABORTING!!!
Number of WU records:  0
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
2019-05-29  --test --verbose | (head -n 11 ; tail -n 3)
Using configuration

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Thanks, Rod!  =D

Those are precisely the same tests I ran and exact same results that noted, 
before  before publishing.  =D
Plus a ton of checks against my own station, KMIDEARB5, west of UTC.

As before, even more important than these 'historical' API tests is to query 
against the current day at different times throughout the day, such as before 
and after your midnight and before and after UTC midnight.  That logic which 
uses the 'today' API (rapid) for today is the one we will likely keep no matter 
what IBM does with the well-known bugs in the 'historical' API.

(I can't test the 'today' logic 100% without changing my localtime on my system 
and I don't want to do that, which is why I appreciate help from others to 
validate the code).  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Jun 3, 2019, at 5:39 PM, Rod Yager  wrote:
> 
> Dear Leon,
> 
> Thanks for all your work on this. This works as well as we are going to be 
> able to manage unless and until WU fixes the bug. 
> 
> The remaining issue affects just one date, the  “change-over” date when WU 
> transitions to returning the correct data - and will only bite for stations 
> east of UTC, and only in the hours that they are ahead of UTC.
> 
> Currently, for me, that date is 2019-05-28. When it originally asks for 
> 2019-05-28, it returns the data for 2019-05-29. Wunderfixer then tries to 
> compensate by asking for 2019-05-27. But 2019-05-27 is a date before the WU 
> bug, and so it returns the data for 2019-05-27. We can’t work around this, 
> because no request to WU will actually return the data we want. The only fix 
> is to wait a few hours until the UTC date is aligned with the Sydney date.
> 
> Good news is that for stations west of UTC, this won’t happen as the bug 
> causes a duplicate day, not a missing day.
> 
> Here’s the output for the requests for my station from 2019-05-27 (before the 
> WU bug kicks in) to today 2019-06-04.
> 
> 
> [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
> 2019-05-27  --test --verbose | (head -n 11 ; tail -n 3)
> Using configuration file /home/weewx/weewx.conf.
> Weather Underground Station:   ISYDNEY155
> Date to check: 2019-05-27
> WU API obsTimeLocal:   2019-05-27
> epoch: 1558879200 date_epoch_local: 2019-05-27 00:00:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:00:00Z obsTimeLocal: 2019-05-27 00:00:00
> epoch: 1558879500 date_epoch_local: 2019-05-27 00:05:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:05:00Z obsTimeLocal: 2019-05-27 00:05:00
> epoch: 1558879800 date_epoch_local: 2019-05-27 00:10:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:10:00Z obsTimeLocal: 2019-05-27 00:10:00
> epoch: 1558880100 date_epoch_local: 2019-05-27 00:15:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:15:00Z obsTimeLocal: 2019-05-27 00:15:00
> epoch: 1558880400 date_epoch_local: 2019-05-27 00:20:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:20:00Z obsTimeLocal: 2019-05-27 00:20:00
> epoch: 1558880700 date_epoch_local: 2019-05-27 00:25:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:25:00Z obsTimeLocal: 2019-05-27 00:25:00
> epoch: 1558881000 date_epoch_local: 2019-05-27 00:30:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:30:00Z obsTimeLocal: 2019-05-27 00:30:00
> epoch: 1558965000 date_epoch_local: 2019-05-27 23:50:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-27T13:50:00Z obsTimeLocal: 2019-05-27 23:50:00
> epoch: 1558965300 date_epoch_local: 2019-05-27 23:55:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-27T13:55:00Z obsTimeLocal: 2019-05-27 23:55:00
> Number of WU records:  288
> [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
> 2019-05-28  --test --verbose | (head -n 11 ; tail -n 3)
> 
> No results returned from Weather Underground (perhaps a bad station name??).
> Using configuration file /home/weewx/weewx.conf.
> Weather Underground Station:   ISYDNEY155
> Date to check: 2019-05-28
> WU API obsTimeLocal:   2019-05-29
> WU API RETURNED WRONG DATE!!!
> WU API COMPENSATION DATE:  2019-05-27
> WU API obsTimeLocal:   2019-05-27
> WU API COMPENSATION FAILURE! ABORTING!!!
> Number of WU records:  0
> [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
> 2019-05-29  --test --verbose | (head -n 11 ; tail -n 3)
> Using configuration file /home/weewx/weewx.conf.
> Weather Underground Station:   ISYDNEY155
> Date to check: 2019-05-29
> WU API obsTimeLocal:   2019-05-30
> WU API RETURNED WRONG DATE!!!
> WU API COMPENSATION DATE:  2019-05-28
> WU API obsTimeLocal:   2019-05-29
> epoch: 1559052000 date_epoch_local: 2019-05-29 00:00:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-28T14:00:00Z obsTimeLocal: 2019-05-29 00:00:00
> epoch: 1559052300 date_epoch_local: 2019-05-29 00:05:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-28T14:05:00Z obsTimeLocal: 2019-05-

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Rod Yager]

Dear Leon,

Thanks for all your work on this. This works as well as we are going to be able 
to manage unless and until WU fixes the bug.

The remaining issue affects just one date, the  “change-over” date when WU 
transitions to returning the correct data - and will only bite for stations 
east of UTC, and only in the hours that they are ahead of UTC.

Currently, for me, that date is 2019-05-28. When it originally asks for 
2019-05-28, it returns the data for 2019-05-29. Wunderfixer then tries to 
compensate by asking for 2019-05-27. But 2019-05-27 is a date before the WU 
bug, and so it returns the data for 2019-05-27. We can’t work around this, 
because no request to WU will actually return the data we want. The only fix is 
to wait a few hours until the UTC date is aligned with the Sydney date.

Good news is that for stations west of UTC, this won’t happen as the bug causes 
a duplicate day, not a missing day.

Here’s the output for the requests for my station from 2019-05-27 (before the 
WU bug kicks in) to today 2019-06-04.


[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
2019-05-27  --test --verbose | (head -n 11 ; tail -n 3)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-05-27
WU API obsTimeLocal:   2019-05-27
epoch: 1558879200 date_epoch_local: 2019-05-27 00:00:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:00:00Z obsTimeLocal: 2019-05-27 00:00:00
epoch: 1558879500 date_epoch_local: 2019-05-27 00:05:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:05:00Z obsTimeLocal: 2019-05-27 00:05:00
epoch: 1558879800 date_epoch_local: 2019-05-27 00:10:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:10:00Z obsTimeLocal: 2019-05-27 00:10:00
epoch: 1558880100 date_epoch_local: 2019-05-27 00:15:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:15:00Z obsTimeLocal: 2019-05-27 00:15:00
epoch: 1558880400 date_epoch_local: 2019-05-27 00:20:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:20:00Z obsTimeLocal: 2019-05-27 00:20:00
epoch: 1558880700 date_epoch_local: 2019-05-27 00:25:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:25:00Z obsTimeLocal: 2019-05-27 00:25:00
epoch: 1558881000 date_epoch_local: 2019-05-27 00:30:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-26T14:30:00Z obsTimeLocal: 2019-05-27 00:30:00
epoch: 1558965000 date_epoch_local: 2019-05-27 23:50:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-27T13:50:00Z obsTimeLocal: 2019-05-27 23:50:00
epoch: 1558965300 date_epoch_local: 2019-05-27 23:55:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-27T13:55:00Z obsTimeLocal: 2019-05-27 23:55:00
Number of WU records:  288
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
2019-05-28  --test --verbose | (head -n 11 ; tail -n 3)

No results returned from Weather Underground (perhaps a bad station name??).
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-05-28
WU API obsTimeLocal:   2019-05-29
WU API RETURNED WRONG DATE!!!
WU API COMPENSATION DATE:  2019-05-27
WU API obsTimeLocal:   2019-05-27
WU API COMPENSATION FAILURE! ABORTING!!!
Number of WU records:  0
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
2019-05-29  --test --verbose | (head -n 11 ; tail -n 3)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-05-29
WU API obsTimeLocal:   2019-05-30
WU API RETURNED WRONG DATE!!!
WU API COMPENSATION DATE:  2019-05-28
WU API obsTimeLocal:   2019-05-29
epoch: 1559052000 date_epoch_local: 2019-05-29 00:00:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-28T14:00:00Z obsTimeLocal: 2019-05-29 00:00:00
epoch: 1559052300 date_epoch_local: 2019-05-29 00:05:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-28T14:05:00Z obsTimeLocal: 2019-05-29 00:05:00
epoch: 1559052600 date_epoch_local: 2019-05-29 00:10:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-28T14:10:00Z obsTimeLocal: 2019-05-29 00:10:00
epoch: 1559052900 date_epoch_local: 2019-05-29 00:15:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-28T14:15:00Z obsTimeLocal: 2019-05-29 00:15:00
epoch: 1559137800 date_epoch_local: 2019-05-29 23:50:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-29T13:50:00Z obsTimeLocal: 2019-05-29 23:50:00
epoch: 1559138100 date_epoch_local: 2019-05-29 23:55:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-29T13:55:00Z obsTimeLocal: 2019-05-29 23:55:00
Number of WU records:  288
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
2019-05-30  --test --verbose | (head -n 11 ; tail -n 3)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-05-30
WU API obsTimeLocal:   2019-05-31
WU API RETURNED WRONG DATE!!!
WU API COMPENSATION DATE:  2019-05-29
WU API obs

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

All,
There are updates to the wunderdates utilities, to implement the following:

Add +/- 1-day compensation logic to workaround WU 'historical' API bug, 
depending on whether station is east/west of UTC, and whether it is within X 
hours of UTC midnight, relative to the local station time.

These wunderdates utilities are for debug purposes only to aid in testing the 
WU behavior, while IBM works to fix the actual bug (no intention of adding this 
logic to wunderfixer, which will remain broken until IBM fixes their API).

The bugs are well known / well documented.
The continued use of wunderdates is just to keep an eye on IBM's progress in 
fixing the bugs, e.g. keep an eye out for if the behavior changes / improves.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Jun 1, 2019, at 1:17 PM, Leon Shaner  wrote:
> 
> All,
> 
> I'm testing a new approach.  Below you will find links to an updated 
> wunderdates utility that can be used to validate whether I am on the right 
> track.
> The wunderdates utility simply dumps out timestamp related records from what 
> WU returns from the query.
> 
> We're looking mainly at the obsTimeUtc and obsTimeLocal values, which were 
> demonstrated under the prior approach to be returning the wrong dates when 
> within the stations localtime +/- offset from UTC.
> 
> The new approach is to detect if the requested date is 'today' and if so, use 
> a different API URL that already assumes 'today' and will hopefully not be 
> subject to the UTC offset bugs we've been chasing with the historical data 
> API URL.
> 
> I have my crontab set to do another test approaching my UTC offset, just 
> after coming within the offset, and then again just before and after midnight 
> localtime.
> (Same test I did before, but now with the new approach in place).
> 
> Here is the utility for anyone else that wants to check out the behavior:
> 
> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3
> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4
> 
> Which version you pull will depend on which base weewx you are running.
> Pull the one that matches your weewx version and place it in bin, next to 
> wunderfixer, and it will take the same arguments as wunderfixer.
> 
> You can just try wunderfixer as you normally would (with --apikey) and then 
> run wunderdates(3 or 4) with exact same arguments to be able to see what 
> actual timestamps WU is sending back for the date queried.
> Parameters like --epsilon don't have any effect in the case of wunderdates, 
> but I left it there so you don't have to change options when running the util 
> to get the debug output.
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 30, 2019, at 12:40 AM, Leon Shaner  wrote:
>> 
>> Rod,
>> Thanks again for this.
>> 
>> Since the in-progress version of wunderfixer doesn't really show you the 
>> dates that come back from WU, I have written a tool just to debug the dates.
>> 
>> The command-line input and basing of defaults on weewx.conf works the same 
>> as wunderfixer, except it doesn't look at your DB at all.  It only prints 
>> out datestamps in various incarnations coming back from WU and as compared 
>> to your system's localtime.
>> 
>> It would be helpful to see the "wunderdates" output at times like you've 
>> shown below, a la before and after your localtime rolls around to UTC 
>> midnight.
>> 
>> Since you are at UTC + 10, another interesting time would be 11+ hours on 
>> either side of UTC midnight, in addition to within 9 or less hours.  This is 
>> just to make sure we're covering all the corner cases.
>> 
>> Gary reported a difference for stations that are east vs. west of GMT, and I 
>> expect we're really chasing the same bug in that there is some bad math WU 
>> is doing based on UTC offset, but since an offset can be +/-, the effects go 
>> in opposite directions date-wise, depending on which side of the UTC 
>> dateline your station is located.
>> 
>> At least that is what I surmise may be happening, but the wunderdates 
>> utility should shed light one way or the other.  =D
>> 
>> The wunderdates utility is available at the links below.
>> 
>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3
>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4
>> 
>> Which version you pull will depend on which base weewx you are running.
>> Pull the one that matches your weewx version and place it in bin, next to 
>> wunderfixer, and it will take the same arguments as wunderfixer.
>> 
>> You can just try wunderfixer as you normally would (with --apikey) and then 
>> run wunderdates(3 or 4) with exact same arguments to be able to see what 
>> actual timestamps WU is sending back for the date queried.
>> Parameters like --epsilon don't have any effect in the case of wunderdates, 
>> but I left it there so you don't have to change options when running the 
>> util to get the debug output.
>> 
>

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Rod,

I've spent quite a bit of time working on a "cleaner" workaround for the WU 
bugs.
But just when I thought I had landed on the "reasonable" approach, I hit a snag.

As a reminder, with the latest approach, all is fine for stations west of the 
UTC date line.   What I am working on now is a workaround for stations east of 
the UTC date line, where the data returned is a day after the date requested.

The wall that I just hit is that the WU behavior around this latest bug 
changes, when looking historically.  :-(

>From today's date going back to 2019-05-28, the query returns obsTimeLocal 
>data from the day after the date requested.  This is the bug we are trying to 
>workaround.
To compensate, I can ask for the prior date to get the date we really want.

HOWEVER, starting with 2019-05-26 (and prior), the query returns obsTimeLocal 
data for the date that was actually requested.

This leads to the inability to get data for 2019-05-27, because the actual 
query for that date returns 2019-05-28 datum, and the compensation date of 
2019-05-26 returns actual data for 2019-05-26.

I think I am done trying to work around this bug.  IBM needs to own it.

See here -- scroll to second to last example where 2019-05-27 is requested, but 
when I tried 2019-05-26, I got back data for the actual 2019-05-26, so 
compensation did not work.  :-(

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates 
--epsilon=125 --date 2019-06-02 --station ISYDNEY155 --test --verbose | (head 
-n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-06-02
WU API obsTimeLocal:   2019-06-03
WU API WRONG DATE !!!
WU API COMPENSATION DATE:  2019-06-01
WU API obsTimeLocal:   2019-06-02
epoch: 1559397600 date_epoch_local: 2019-06-01 10:00:00 tz: Australia/Sydney 
obsTimeUtc: 2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 00:00:00
epoch: 1559397900 date_epoch_local: 2019-06-01 10:05:00 tz: Australia/Sydney 
obsTimeUtc: 2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 00:05:00
Number of WU records:  288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates 
--epsilon=125 --date 2019-06-01 --station ISYDNEY155 --test --verbose | (head 
-n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-06-01
WU API obsTimeLocal:   2019-06-02
WU API WRONG DATE !!!
WU API COMPENSATION DATE:  2019-05-31
WU API obsTimeLocal:   2019-06-01
epoch: 1559311200 date_epoch_local: 2019-05-31 10:00:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-31T14:00:00Z obsTimeLocal: 2019-06-01 00:00:00
epoch: 1559311500 date_epoch_local: 2019-05-31 10:05:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-31T14:05:00Z obsTimeLocal: 2019-06-01 00:05:00
Number of WU records:  288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates 
--epsilon=125 --date 2019-05-31 --station ISYDNEY155 --test --verbose | (head 
-n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-05-31
WU API obsTimeLocal:   2019-06-01
WU API WRONG DATE !!!
WU API COMPENSATION DATE:  2019-05-30
WU API obsTimeLocal:   2019-05-31
epoch: 1559224800 date_epoch_local: 2019-05-30 10:00:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T14:00:00Z obsTimeLocal: 2019-05-31 00:00:00
epoch: 1559225100 date_epoch_local: 2019-05-30 10:05:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T14:05:00Z obsTimeLocal: 2019-05-31 00:05:00
Number of WU records:  288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates 
--epsilon=125 --date 2019-05-30 --station ISYDNEY155 --test --verbose | (head 
-n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-05-30
WU API obsTimeLocal:   2019-05-31
WU API WRONG DATE !!!
WU API COMPENSATION DATE:  2019-05-29
WU API obsTimeLocal:   2019-05-30
epoch: 1559138400 date_epoch_local: 2019-05-29 10:00:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-29T14:00:00Z obsTimeLocal: 2019-05-30 00:00:00
epoch: 1559138700 date_epoch_local: 2019-05-29 10:05:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-29T14:05:00Z obsTimeLocal: 2019-05-30 00:05:00
Number of WU records:  288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates 
--epsilon=125 --date 2019-05-29 --station ISYDNEY155 --test --verbose | (head 
-n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-05-29
WU API obsTimeLocal:   2019-05-30
WU API WRONG DATE !!!
WU API COMPENSATION DATE:  2019-05-28
WU API obsTimeLocal:

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Rod,

It's interesting that the "workaround" that I've already done, is actually 
working 100% for me.   I got good results when combining the 'today' API when 
asking for the current date, and only using the 'historical' result when asking 
for dates other than the current date -- relative to station local time.

Here is my KMIDEARB5 station's transition from 7:59 p.m. to 8:00 p.m, where 
time moves to within the UTC-4 offset:

$ diff 59:19:1:6_wu.txt 0:20:1:6_wu.txt
241c241,242
< Number of WU records:  239
---
> epoch: 1559433599 date_epoch_local: 2019-06-01 19:59:59 utcepoch: 1559433999 
> date_epoch_utc: 2019-06-01 23:59:59 tz: America/New_York obsTimeUtc: 
> 2019-06-01T23:59:59Z obsTimeLocal: 2019-06-01 19:59:59
> Number of WU records:  240

The above is exactly correct -- the second query is one minute later and is 
after/on the 5-minute record "boundary" hence one additional record.

And then as my station transition from 11:59 p.m. to midnight next day.  Before 
midnight the 'today' API is used, but after midnight the 'historical' API will 
be used, which was always working for me after midnight (station local time):

$ tail -n 4 59:23:1:6_wu.txt 0:0:2:6_wu.txt
==> 59:23:1:6_wu.txt <==
epoch: 1559447399 date_epoch_local: 2019-06-01 23:49:59 utcepoch: 1559447799 
date_epoch_utc: 2019-06-02 03:49:59 tz: America/New_York obsTimeUtc: 
2019-06-02T03:49:59Z obsTimeLocal: 2019-06-01 23:49:59
epoch: 1559447699 date_epoch_local: 2019-06-01 23:54:59 utcepoch: 1559448099 
date_epoch_utc: 2019-06-02 03:54:59 tz: America/New_York obsTimeUtc: 
2019-06-02T03:54:59Z obsTimeLocal: 2019-06-01 23:54:59
epoch: 1559447701 date_epoch_local: 2019-06-01 23:55:01 utcepoch: 1559448101 
date_epoch_utc: 2019-06-02 03:55:01 tz: America/New_York obsTimeUtc: 
2019-06-02T03:55:01Z obsTimeLocal: 2019-06-01 23:55:01
Number of WU records:  288

==> 0:0:2:6_wu.txt <==
epoch: 1559447399 date_epoch_local: 2019-06-01 23:49:59 utcepoch: 1559447799 
date_epoch_utc: 2019-06-02 03:49:59 tz: America/New_York obsTimeUtc: 
2019-06-02T03:49:59Z obsTimeLocal: 2019-06-01 23:49:59
epoch: 1559447699 date_epoch_local: 2019-06-01 23:54:59 utcepoch: 1559448099 
date_epoch_utc: 2019-06-02 03:54:59 tz: America/New_York obsTimeUtc: 
2019-06-02T03:54:59Z obsTimeLocal: 2019-06-01 23:54:59
epoch: 1559447999 date_epoch_local: 2019-06-01 23:59:59 utcepoch: 1559448399 
date_epoch_utc: 2019-06-02 03:59:59 tz: America/New_York obsTimeUtc: 
2019-06-02T03:59:59Z obsTimeLocal: 2019-06-01 23:59:59
Number of WU records:  288

There again, the second query is a minute later, so one more result, but with 
explanation...  ;-)

There is a slight difference between the 'today' and 'historical' result.
The first result, using the 'today' API ran at 1 minute before midnight, which 
was on the cusp of posting the 1-minute record nearest actual midnight.
The 'today' result shows two time-stamps very close to 11:55 p.m.
Where-as the 'historical' result replaces the later ~11:55 p.m. result with the 
~midnight result.

Those differences are "in the noise" and inconsequential.  Even the default 
wunderfixer --epsilon=120 will filter out such a duplicate.

I am happy that this new approach with the combined 'today' and 'historical' 
queries is at least effective for stations west of the UTC dateline.
It means I have easily worked around half of the WU API bug.

Now to further press IBM to fix the other half of the bug...  =D
(Even while I continue to think about ways to *cleanly* workaround the bug for 
stations east of UTC dateline).

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Jun 1, 2019, at 11:08 PM, Rod Yager  wrote:
> 
> Dear Leon,
> 
> If WU doesn’t fix their bug, the solution is really quite simple.
> 
> We need to make 3 requests to WU for the data: one request for the date we 
> are interested in, one request for the day before and one request for the day 
> after. 
> 
> eg. If we are intending to upload missing data for 2019-01-01 we need to ask 
> WU for the records it has for 2018-12-31, 2019-01-01 and 2019-01-02. 
> 
> One of these will have the information pertaining to the day we are 
> interested in.  
> 
> We then need to filter the three responses from WU, only keeping the 
> responses that pertain to the day we are interested in.
> 
> At this point, we have exactly what we want from WU and so we compare that 
> with the local database, and upload the data that is missing in the WU 
> record, just as we always did.
> 
> 
> in your timezone (UTC-4) if you ask for data for your station the correct 
> information will be in 
> the same day request:  if the request is made between 00:00 and 19:59   (at 
> those times, the UTC date is the same as the date where you are )
> the request for the next day: if the request is made between 20:00 and 23:59  
> (at those times, the UTC date is one day ahead the date where you are)
> 
> If you request information for my station (I’ve sent you the details in a 
> PM), th

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Rod Yager]

Dear Leon,

If WU doesn’t fix their bug, the solution is really quite simple.

We need to make 3 requests to WU for the data: one request for the date we are 
interested in, one request for the day before and one request for the day after.

eg. If we are intending to upload missing data for 2019-01-01 we need to ask WU 
for the records it has for 2018-12-31, 2019-01-01 and 2019-01-02.

One of these will have the information pertaining to the day we are interested 
in.

We then need to filter the three responses from WU, only keeping the responses 
that pertain to the day we are interested in.

At this point, we have exactly what we want from WU and so we compare that with 
the local database, and upload the data that is missing in the WU record, just 
as we always did.


in your timezone (UTC-4) if you ask for data for your station the correct 
information will be in

  *   the same day request:  if the request is made between 00:00 and 19:59   
(at those times, the UTC date is the same as the date where you are )
  *   the request for the next day: if the request is made between 20:00 and 
23:59  (at those times, the UTC date is one day ahead the date where you are)

If you request information for my station (I’ve sent you the details in a PM), 
the correct information will be in

  *   the same day request: if the request is made between 10:00 and 19:59 your 
time  (at those times,  the UTC date is the same as the date where my station 
is located)
  *   the request for the previous day: if the request is made between 0:00 and 
09:59 or between 20:00 and 23:59 your time (at those times, the UTC date is one 
day behind the date where my station is located)

Rod



On 2 Jun 2019, at 9:27 am, Leon Shaner 
mailto:l...@isylum.org>> wrote:

Rod,

Thanks for testing.  Well, WU's approach is flawed, not mine, really.  :S

It means that both API's, the one I just added for 'today" and the one from 
before that was for 'historical' (including today), both have the same bug.

For now, the best workaround I can think of is to only ever run the new 
wunderfixer only ever for the previous day, only ever after UTC has rolled over 
midnight.   :-(
E.g. when it is 2019-06-02 in UTC, you can get the right results with 
wunderfixer if you ask for 2019-06-01 only.In other words, always delay 
your wunderfixer fix ups by a day, operating always on yesterday's date 
(relative to UTC) after UTC has rolled over to the next day.

Scroll to see the output of me running wunderdates on my local machine in UTC-4 
against your station in UTC+10.

Because I am asking for 2019-06-01, which is also my local date, the logic will 
use the URL that queries for 'today' and instead of returning obsTimeLocal 
records, it is returning obsTimeUtc records for the requested date.  We can see 
that it's returning 2019-06-02 relative to your station, even though I asked 
for 2019-06-01.

Sure, I could go a bit further and use a related API query to check the 
timezone of your station and use date conversions to check if the date 
requested is 'today' relative to the station requested, but really this is 
intended to be run on the same machine as weewx, so the localtime date should 
match that of the station.  It means there isn't any real reason for me to 
workaround their bug.  In fact even if I did, then the logic would just fall 
through to the historical data URL, which we already know is buggy.

For now, I'll just report this to IBM a bug in the 'today' API, similar to the 
bug in the 'historical' API.

$ ./wunderdates --epsilon=125 --date 2019-06-01 --station ISYDNEY155 --test 
--verbose | more

Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-06-01
epoch: 1559397600 date_epoch_local: 2019-06-01 10:00:00 utcepoch: 1559398000 
date_epoch_utc: 2019-06-01 14:00:00 tz: Australia/Sydney obsTimeUtc: 
2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 00:00:00
epoch: 1559397900 date_epoch_local: 2019-06-01 10:05:00 utcepoch: 1559398300 
date_epoch_utc: 2019-06-01 14:05:00 tz: Australia/Sydney obsTimeUtc: 
2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 00:05:00
...


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

On Jun 1, 2019, at 6:03 PM, Rod Yager 
mailto:r...@yager.net.au>> wrote:

Dear Leon,

Your approach is flawed. It won’t work for historical data.

If you run

wunderdates —date=2019-01-01   at 9pm your time   you will receive the data for 
2018-12-31  (because your station is a day behind UTC at the time of the 
request, so it gives you the data for the previous day).

On the other hand, if you run

wunderdates —date=2019-01-01   at 9am your time   you will receive the data for 
2019-01-01  (because your station is in the same day as UTC at the time of the 
request).

If I do the same thing (I’m in UTC+10)

wunderdates —date=2019-01-01   at 9pm my time  produces the data for 2019-01-01 
(because my station is in the same day as UTC at the time 

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Rod,

Thanks for testing.  Well, WU's approach is flawed, not mine, really.  :S

It means that both API's, the one I just added for 'today" and the one from 
before that was for 'historical' (including today), both have the same bug.

For now, the best workaround I can think of is to only ever run the new 
wunderfixer only ever for the previous day, only ever after UTC has rolled over 
midnight.   :-(
E.g. when it is 2019-06-02 in UTC, you can get the right results with 
wunderfixer if you ask for 2019-06-01 only.In other words, always delay 
your wunderfixer fix ups by a day, operating always on yesterday's date 
(relative to UTC) after UTC has rolled over to the next day.

Scroll to see the output of me running wunderdates on my local machine in UTC-4 
against your station in UTC+10.

Because I am asking for 2019-06-01, which is also my local date, the logic will 
use the URL that queries for 'today' and instead of returning obsTimeLocal 
records, it is returning obsTimeUtc records for the requested date.  We can see 
that it's returning 2019-06-02 relative to your station, even though I asked 
for 2019-06-01.

Sure, I could go a bit further and use a related API query to check the 
timezone of your station and use date conversions to check if the date 
requested is 'today' relative to the station requested, but really this is 
intended to be run on the same machine as weewx, so the localtime date should 
match that of the station.  It means there isn't any real reason for me to 
workaround their bug.  In fact even if I did, then the logic would just fall 
through to the historical data URL, which we already know is buggy.

For now, I'll just report this to IBM a bug in the 'today' API, similar to the 
bug in the 'historical' API.

$ ./wunderdates --epsilon=125 --date 2019-06-01 --station ISYDNEY155 --test 
--verbose | more

Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-06-01
epoch: 1559397600 date_epoch_local: 2019-06-01 10:00:00 utcepoch: 1559398000 
date_epoch_utc: 2019-06-01 14:00:00 tz: Australia/Sydney obsTimeUtc: 
2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 00:00:00
epoch: 1559397900 date_epoch_local: 2019-06-01 10:05:00 utcepoch: 1559398300 
date_epoch_utc: 2019-06-01 14:05:00 tz: Australia/Sydney obsTimeUtc: 
2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 00:05:00
...


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Jun 1, 2019, at 6:03 PM, Rod Yager  wrote:
> 
> Dear Leon,
> 
> Your approach is flawed. It won’t work for historical data.
> 
> If you run
> 
> wunderdates —date=2019-01-01   at 9pm your time   you will receive the data 
> for 2018-12-31  (because your station is a day behind UTC at the time of the 
> request, so it gives you the data for the previous day).
> 
> On the other hand, if you run 
> 
> wunderdates —date=2019-01-01   at 9am your time   you will receive the data 
> for 2019-01-01  (because your station is in the same day as UTC at the time 
> of the request).
> 
> If I do the same thing (I’m in UTC+10)
> 
> wunderdates —date=2019-01-01   at 9pm my time  produces the data for 
> 2019-01-01 (because my station is in the same day as UTC at the time of the 
> request)  but
> wunderdates —date=2019-01-01   at 9am my time produces the data for 
> 2019-01-02 (because my station is a day ahead of UTC at the time of the 
> request) 
> 
> Rod
> 
>> On 2 Jun 2019, at 3:17 am, Leon Shaner  wrote:
>> 
>> All,
>> 
>> I'm testing a new approach.  Below you will find links to an updated 
>> wunderdates utility that can be used to validate whether I am on the right 
>> track.
>> The wunderdates utility simply dumps out timestamp related records from what 
>> WU returns from the query.
>> 
>> We're looking mainly at the obsTimeUtc and obsTimeLocal values, which were 
>> demonstrated under the prior approach to be returning the wrong dates when 
>> within the stations localtime +/- offset from UTC.
>> 
>> The new approach is to detect if the requested date is 'today' and if so, 
>> use a different API URL that already assumes 'today' and will hopefully not 
>> be subject to the UTC offset bugs we've been chasing with the historical 
>> data API URL.
>> 
>> I have my crontab set to do another test approaching my UTC offset, just 
>> after coming within the offset, and then again just before and after 
>> midnight localtime.
>> (Same test I did before, but now with the new approach in place).
>> 
>> Here is the utility for anyone else that wants to check out the behavior:
>> 
>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3
>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4
>> 
>> Which version you pull will depend on which base weewx you are running.
>> Pull the one that matches your weewx version and place it in bin, next to 
>> wunderfixer, and it will take the same arguments as wunderfixer.
>> 
>> You can just try wunderfixer as you normall

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Rod Yager]

Dear Leon,

Your approach is flawed. It won’t work for historical data.

If you run

wunderdates —date=2019-01-01   at 9pm your time   you will receive the data for 
2018-12-31  (because your station is a day behind UTC at the time of the 
request, so it gives you the data for the previous day).

On the other hand, if you run

wunderdates —date=2019-01-01   at 9am your time   you will receive the data for 
2019-01-01  (because your station is in the same day as UTC at the time of the 
request).

If I do the same thing (I’m in UTC+10)

wunderdates —date=2019-01-01   at 9pm my time  produces the data for 2019-01-01 
(because my station is in the same day as UTC at the time of the request)  but
wunderdates —date=2019-01-01   at 9am my time produces the data for 2019-01-02 
(because my station is a day ahead of UTC at the time of the request)

Rod

On 2 Jun 2019, at 3:17 am, Leon Shaner 
mailto:l...@isylum.org>> wrote:

All,

I'm testing a new approach.  Below you will find links to an updated 
wunderdates utility that can be used to validate whether I am on the right 
track.
The wunderdates utility simply dumps out timestamp related records from what WU 
returns from the query.

We're looking mainly at the obsTimeUtc and obsTimeLocal values, which were 
demonstrated under the prior approach to be returning the wrong dates when 
within the stations localtime +/- offset from UTC.

The new approach is to detect if the requested date is 'today' and if so, use a 
different API URL that already assumes 'today' and will hopefully not be 
subject to the UTC offset bugs we've been chasing with the historical data API 
URL.

I have my crontab set to do another test approaching my UTC offset, just after 
coming within the offset, and then again just before and after midnight 
localtime.
(Same test I did before, but now with the new approach in place).

Here is the utility for anyone else that wants to check out the behavior:

https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3
https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4

Which version you pull will depend on which base weewx you are running.
Pull the one that matches your weewx version and place it in bin, next to 
wunderfixer, and it will take the same arguments as wunderfixer.

You can just try wunderfixer as you normally would (with --apikey) and then run 
wunderdates(3 or 4) with exact same arguments to be able to see what actual 
timestamps WU is sending back for the date queried.
Parameters like --epsilon don't have any effect in the case of wunderdates, but 
I left it there so you don't have to change options when running the util to 
get the debug output.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

On May 30, 2019, at 12:40 AM, Leon Shaner 
mailto:l...@isylum.org>> wrote:

Rod,
Thanks again for this.

Since the in-progress version of wunderfixer doesn't really show you the dates 
that come back from WU, I have written a tool just to debug the dates.

The command-line input and basing of defaults on weewx.conf works the same as 
wunderfixer, except it doesn't look at your DB at all.  It only prints out 
datestamps in various incarnations coming back from WU and as compared to your 
system's localtime.

It would be helpful to see the "wunderdates" output at times like you've shown 
below, a la before and after your localtime rolls around to UTC midnight.

Since you are at UTC + 10, another interesting time would be 11+ hours on 
either side of UTC midnight, in addition to within 9 or less hours.  This is 
just to make sure we're covering all the corner cases.

Gary reported a difference for stations that are east vs. west of GMT, and I 
expect we're really chasing the same bug in that there is some bad math WU is 
doing based on UTC offset, but since an offset can be +/-, the effects go in 
opposite directions date-wise, depending on which side of the UTC dateline your 
station is located.

At least that is what I surmise may be happening, but the wunderdates utility 
should shed light one way or the other.  =D

The wunderdates utility is available at the links below.

https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3
https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4

Which version you pull will depend on which base weewx you are running.
Pull the one that matches your weewx version and place it in bin, next to 
wunderfixer, and it will take the same arguments as wunderfixer.

You can just try wunderfixer as you normally would (with --apikey) and then run 
wunderdates(3 or 4) with exact same arguments to be able to see what actual 
timestamps WU is sending back for the date queried.
Parameters like --epsilon don't have any effect in the case of wunderdates, but 
I left it there so you don't have to change options when running the util to 
get the debug output.

What I've been doing is saving the output to files for use with sdiff 
(side-by-side) diff.
Or you can just compare t

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

All,

I'm testing a new approach.  Below you will find links to an updated 
wunderdates utility that can be used to validate whether I am on the right 
track.
The wunderdates utility simply dumps out timestamp related records from what WU 
returns from the query.

We're looking mainly at the obsTimeUtc and obsTimeLocal values, which were 
demonstrated under the prior approach to be returning the wrong dates when 
within the stations localtime +/- offset from UTC.

The new approach is to detect if the requested date is 'today' and if so, use a 
different API URL that already assumes 'today' and will hopefully not be 
subject to the UTC offset bugs we've been chasing with the historical data API 
URL.

I have my crontab set to do another test approaching my UTC offset, just after 
coming within the offset, and then again just before and after midnight 
localtime.
(Same test I did before, but now with the new approach in place).

Here is the utility for anyone else that wants to check out the behavior:

https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3
https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4

Which version you pull will depend on which base weewx you are running.
Pull the one that matches your weewx version and place it in bin, next to 
wunderfixer, and it will take the same arguments as wunderfixer.

You can just try wunderfixer as you normally would (with --apikey) and then run 
wunderdates(3 or 4) with exact same arguments to be able to see what actual 
timestamps WU is sending back for the date queried.
Parameters like --epsilon don't have any effect in the case of wunderdates, but 
I left it there so you don't have to change options when running the util to 
get the debug output.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 30, 2019, at 12:40 AM, Leon Shaner  wrote:
> 
> Rod,
> Thanks again for this.
> 
> Since the in-progress version of wunderfixer doesn't really show you the 
> dates that come back from WU, I have written a tool just to debug the dates.
> 
> The command-line input and basing of defaults on weewx.conf works the same as 
> wunderfixer, except it doesn't look at your DB at all.  It only prints out 
> datestamps in various incarnations coming back from WU and as compared to 
> your system's localtime.
> 
> It would be helpful to see the "wunderdates" output at times like you've 
> shown below, a la before and after your localtime rolls around to UTC 
> midnight.
> 
> Since you are at UTC + 10, another interesting time would be 11+ hours on 
> either side of UTC midnight, in addition to within 9 or less hours.  This is 
> just to make sure we're covering all the corner cases.
> 
> Gary reported a difference for stations that are east vs. west of GMT, and I 
> expect we're really chasing the same bug in that there is some bad math WU is 
> doing based on UTC offset, but since an offset can be +/-, the effects go in 
> opposite directions date-wise, depending on which side of the UTC dateline 
> your station is located.
> 
> At least that is what I surmise may be happening, but the wunderdates utility 
> should shed light one way or the other.  =D
> 
> The wunderdates utility is available at the links below.
> 
> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3
> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4
> 
> Which version you pull will depend on which base weewx you are running.
> Pull the one that matches your weewx version and place it in bin, next to 
> wunderfixer, and it will take the same arguments as wunderfixer.
> 
> You can just try wunderfixer as you normally would (with --apikey) and then 
> run wunderdates(3 or 4) with exact same arguments to be able to see what 
> actual timestamps WU is sending back for the date queried.
> Parameters like --epsilon don't have any effect in the case of wunderdates, 
> but I left it there so you don't have to change options when running the util 
> to get the debug output.
> 
> What I've been doing is saving the output to files for use with sdiff 
> (side-by-side) diff.
> Or you can just compare the head and tail of each file individually.
> 
> Example for my system before and after my local time on 5-28 once it was 
> at/after 8 p.m. here, which is within 4 hours of UTC midnight (I am at UTC-4).
> 
> This output is optimized for screen widths 203 or wider.  Sorry.  :S
> Mainly the last two data fields in the output tell what we need to know.
> 
> 
> pi@nixie:/var/tmp $ head -n 3 59:19:28:5_wu.txt  0:20:28:5_wu.txt
> ==> 59:19:28:5_wu.txt <==
> Using configuration file /usr/share/weewx4/weewx.conf.
> epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: 1559016699 
> date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York obsTimeUtc: 
> 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59
> epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: 1559016999 
> date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_Y

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Andrew,

We can't use this as a workaround, but what you're suggesting is essentially 
what WU should be doing with the date that we provide in the query URL.
WU should be doing the conversion to station localtime and returning the values 
for that date, relative to the station.
But they seem to be returning the values for the date relative to UTC instead.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 31, 2019, at 1:14 AM, Andrew Milner  
> wrote:
> 
> . so does this just simplify to the fact that in the api the date is 
> taken as being the UTC date?
> ie if one requests data for 20190529, and timezone is +3 you get back data 
> for UTC00:00 29 may to 23:59 29 may 
> which is 03:00 29 may local to 23:59 29 may local AND 00:00 30 may local - 
> 02:59 30 may local
> 
> so to get the data for 29 may local it would be necessary to 
> 1. get data for utc 28 may and discard utc 00:00 - 20:59
> 2. get data for utc 29 may and discard utc 21:00 - 23:59
> 
> or have i not understood correctly?
> 
> 
>> On Friday, 31 May 2019 05:48:09 UTC+3, Rod Yager wrote:
>> 
>> I constructed a cron job which exectued  wunderdates --date=2019-05-30  
>> twice every hour, once 2 minutes before the hour and once 2 minutes after 
>> the hour.
>> 
>> Consistent with my earlier post, between 00:00+10 and 09:59+10 the reply 
>> from the api had data which belonged to 2019-05-31 - the day after the 
>> requested date. Before midnight, and after 10am local time, the reply from 
>> the api had data which belonged to 2019-05-30 (as requested).
>> 
>> I've pasted below some representative samples of the last 3 lines of the 
>> wunderdates output.
>> 
>> Before midnight on May 30: Request is for May 30 data. Replies are May 30 
>> data  Local Date = UTC date.
>> 
>> Thu May 30 20:58:02 2019 UTC+10
>> epoch: 1559213100 date_epoch_local: 2019-05-30 20:45:00 utcepoch: 1559212100 
>> date_epoch_utc: 2019-05-30 10:45:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T10:45:00Z obsTimeLocal: 2019-05-30 20:45:00
>> epoch: 1559213400 date_epoch_local: 2019-05-30 20:50:00 utcepoch: 1559212400 
>> date_epoch_utc: 2019-05-30 10:50:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T10:50:00Z obsTimeLocal: 2019-05-30 20:50:00
>> Number of WU records:  251
>> 
>> 
>> Thu May 30 22:58:03 2019 UTC+10
>> epoch: 155922 date_epoch_local: 2019-05-30 22:40:00 utcepoch: 1559219000 
>> date_epoch_utc: 2019-05-30 12:40:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T12:40:00Z obsTimeLocal: 2019-05-30 22:40:00
>> epoch: 1559220300 date_epoch_local: 2019-05-30 22:45:00 utcepoch: 1559219300 
>> date_epoch_utc: 2019-05-30 12:45:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T12:45:00Z obsTimeLocal: 2019-05-30 22:45:00
>> epoch: 1559220600 date_epoch_local: 2019-05-30 22:50:00 utcepoch: 1559219600 
>> date_epoch_utc: 2019-05-30 12:50:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T12:50:00Z obsTimeLocal: 2019-05-30 22:50:00
>> Number of WU records:  275
>> 
>> 
>> Thu May 30 23:58:02 2019 UTC+10
>> epoch: 1559223900 date_epoch_local: 2019-05-30 23:45:00 utcepoch: 1559222900 
>> date_epoch_utc: 2019-05-30 13:45:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T13:45:00Z obsTimeLocal: 2019-05-30 23:45:00
>> epoch: 1559224200 date_epoch_local: 2019-05-30 23:50:00 utcepoch: 1559223200 
>> date_epoch_utc: 2019-05-30 13:50:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T13:50:00Z obsTimeLocal: 2019-05-30 23:50:00
>> Number of WU records:  287
>> 
>> 
>> 10 hour period after midnight on May 31. Request is for May 30 data. Replies 
>> are May 31 data.  Local Date is 1 day ahead of UTC date
>> 
>> Fri May 31 00:02:02 2019 UTC+10
>> Could not get Weather Underground data.
>> Reason: No JSON object could be decoded
>> Could not get Weather Underground data. Exiting. 
>> 
>> 
>> Fri May 31 00:58:02 2019  UTC+10
>> epoch: 1559227500 date_epoch_local: 2019-05-31 00:45:00 utcepoch: 1559226500 
>> date_epoch_utc: 2019-05-30 14:45:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T14:45:00Z obsTimeLocal: 2019-05-31 00:45:00
>> epoch: 1559227800 date_epoch_local: 2019-05-31 00:50:00 utcepoch: 1559226800 
>> date_epoch_utc: 2019-05-30 14:50:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T14:50:00Z obsTimeLocal: 2019-05-31 00:50:00
>> Number of WU records:  11
>> 
>> 
>> Fri May 31 02:58:03 2019 UTC+10
>> epoch: 1559234700 date_epoch_local: 2019-05-31 02:45:00 utcepoch: 1559233700 
>> date_epoch_utc: 2019-05-30 16:45:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T16:45:00Z obsTimeLocal: 2019-05-31 02:45:00
>> epoch: 1559235000 date_epoch_local: 2019-05-31 02:50:00 utcepoch: 1559234000 
>> date_epoch_utc: 2019-05-30 16:50:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T16:50:00Z obsTimeLocal: 2019-05-31 02:50:00
>> Number of WU records:  35
>> 
>> 
>> Fri May 31 05:58:02 2019 UTC+10
>> epoch: 1559245500 date_epoch_local: 2019-05-31 05:45:00 utcepoch: 1559244500 
>> date_epoch_utc: 2019-05-30 1

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Rod Yager]

No.  You always get all available data for a complete day (00:00:00 to 23:59:59 
local time).

So in UTC+10, the first entry in the returned data is   00:00:00 UTC+10  
(14:00:00 UTC the previous day)  and the last entry in the returned data is 
23:59:59 UTC+10  (13:59:59 UTC  same day)
and in UTC-4, the first entry in the returned data is   00:00:00 UTC-4  
(04:00:00 UTC same day)  and the last entry in the returned data is 23:59:59 
UTC-4  (03:59:59 UTC the  next day)

Consider a request for data for 20190520 made in the UTC+10 timezone.
If the request is made at 01:00 on May 31 local time,   (15:00 May 30 UTC), the 
API will return all data for 20190521. ie. Observations made from 00:00 May 21 
to 23:59 May 21 local time.  (14:00 May 20 UTC to 13:59 May 21 UTC)
If the request is made at 11:00 on May 31 local time,   (01:00 May 31 UTC), the 
API will return all data for 20190520. ie. Observations made from 00:00 May 20 
to 23:59 May 20 local time. (14:00 May 19 UTC to 13:59 May 20 UTC)


I think that the same request for data for 20190520 made in the UTC-4 timezone 
has the following behaviour (judging from what I’ve seen reported on this 
thread)
If the request is made at 11:00 on May 30 local time,   (15:00 May 31 UTC), the 
API will return all data for 20190520. ie. Observations made from 00:00 May 20 
to 23:59 May 20 local time. (04:00 May 20 UTC to 03:59 May 21 UTC)
If the request is made at 21:00 on May 30 local time,   (01:00 May 31 UTC), the 
API will return all data for 20190519. ie. Observations made from 00:00 May 19 
to 23:59 May 19 local time (04:00 May 19 UTC to 03:59 May 20 UTC)


So it appears that Weather Underground is doing the following.

1. Determining whether current date at the station is the same as the current 
date UTC.
2. If the two dates are different, adjust the date in the request to “allow 
for” the date difference.
3. Return the data between 00:00 and 23:59 (local time) on the adjusted date.

Step 2 is unwanted and undesirable.


If Weather Underground doesn’t fix this, wunderfixer needs to work out whether 
the local time date is the same as the UTC date. If it is, send the request as 
usual.
If the local time date is currently ahead of the UTC date, adjust the request 
to ask for the day before the date you actually want.
If the local time date is currently behind the UTC date, adjust the request to 
ask for the day after the date you actually want.


Rod







On 31 May 2019, at 3:14 pm, Andrew Milner 
mailto:andrew.s.r.mil...@gmail.com>> wrote:

. so does this just simplify to the fact that in the api the date is taken 
as being the UTC date?
ie if one requests data for 20190529, and timezone is +3 you get back data for 
UTC00:00 29 may to 23:59 29 may
which is 03:00 29 may local to 23:59 29 may local AND 00:00 30 may local - 
02:59 30 may local

so to get the data for 29 may local it would be necessary to
1. get data for utc 28 may and discard utc 00:00 - 20:59
2. get data for utc 29 may and discard utc 21:00 - 23:59

or have i not understood correctly?


On Friday, 31 May 2019 05:48:09 UTC+3, Rod Yager wrote:

I constructed a cron job which exectued  wunderdates --date=2019-05-30  twice 
every hour, once 2 minutes before the hour and once 2 minutes after the hour.

Consistent with my earlier post, between 00:00+10 and 09:59+10 the reply from 
the api had data which belonged to 2019-05-31 - the day after the requested 
date. Before midnight, and after 10am local time, the reply from the api had 
data which belonged to 2019-05-30 (as requested).

I've pasted below some representative samples of the last 3 lines of the 
wunderdates output.

Before midnight on May 30: Request is for May 30 data. Replies are May 30 data  
Local Date = UTC date.

Thu May 30 20:58:02 2019 UTC+10
epoch: 1559213100 date_epoch_local: 2019-05-30 20:45:00 utcepoch: 1559212100 
date_epoch_utc: 2019-05-30 10:45:00 tz: Australia/Sydney obsTimeUtc: 
2019-05-30T10:45:00Z obsTimeLocal: 2019-05-30 20:45:00
epoch: 1559213400 date_epoch_local: 2019-05-30 20:50:00 utcepoch: 1559212400 
date_epoch_utc: 2019-05-30 10:50:00 tz: Australia/Sydney obsTimeUtc: 
2019-05-30T10:50:00Z obsTimeLocal: 2019-05-30 20:50:00
Number of WU records:  251


Thu May 30 22:58:03 2019 UTC+10
epoch: 155922 date_epoch_local: 2019-05-30 22:40:00 utcepoch: 1559219000 
date_epoch_utc: 2019-05-30 12:40:00 tz: Australia/Sydney obsTimeUtc: 
2019-05-30T12:40:00Z obsTimeLocal: 2019-05-30 22:40:00
epoch: 1559220300 date_epoch_local: 2019-05-30 22:45:00 utcepoch: 1559219300 
date_epoch_utc: 2019-05-30 12:45:00 tz: Australia/Sydney obsTimeUtc: 
2019-05-30T12:45:00Z obsTimeLocal: 2019-05-30 22:45:00
epoch: 1559220600 date_epoch_local: 2019-05-30 22:50:00 utcepoch: 1559219600 
date_epoch_utc: 2019-05-30 12:50:00 tz: Australia/Sydney obsTimeUtc: 
2019-05-30T12:50:00Z obsTimeLocal: 2019-05-30 22:50:00
Number of WU records:  275


Thu May 30 23:58:02 2019 UTC+10
epoch: 1559223900 date_epoch_local: 2019-05-30 23:45:00 utce

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Andrew Milner]

. so does this just simplify to the fact that in the api the date is 
taken as being the UTC date?
ie if one requests data for 20190529, and timezone is +3 you get back data 
for UTC00:00 29 may to 23:59 29 may 
which is 03:00 29 may local to 23:59 29 may local AND 00:00 30 may local - 
02:59 30 may local

so to get the data for 29 may local it would be necessary to 
1. get data for utc 28 may and discard utc 00:00 - 20:59
2. get data for utc 29 may and discard utc 21:00 - 23:59

or have i not understood correctly?


On Friday, 31 May 2019 05:48:09 UTC+3, Rod Yager wrote:
>
>
> I constructed a cron job which exectued  wunderdates --date=2019-05-30 
>  twice every hour, once 2 minutes before the hour and once 2 minutes after 
> the hour.
>
> Consistent with my earlier post, between 00:00+10 and 09:59+10 the reply 
> from the api had data which belonged to 2019-05-31 - the day after the 
> requested date. Before midnight, and after 10am local time, the reply from 
> the api had data which belonged to 2019-05-30 (as requested).
>
> I've pasted below some representative samples of the last 3 lines of the 
> wunderdates output.
>
> Before midnight on May 30: Request is for May 30 data. Replies are May 30 
> data  Local Date = UTC date.
>
> Thu May 30 20:58:02 2019 UTC+10
> epoch: 1559213100 date_epoch_local: 2019-05-30 20:45:00 utcepoch: 
> 1559212100 date_epoch_utc: 2019-05-30 10:45:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T10:45:00Z obsTimeLocal: 2019-05-30 20:45:00
> epoch: 1559213400 date_epoch_local: 2019-05-30 20:50:00 utcepoch: 
> 1559212400 date_epoch_utc: 2019-05-30 10:50:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T10:50:00Z obsTimeLocal: 2019-05-30 20:50:00
> Number of WU records:  251
>
>
> Thu May 30 22:58:03 2019 UTC+10
> epoch: 155922 date_epoch_local: 2019-05-30 22:40:00 utcepoch: 
> 1559219000 date_epoch_utc: 2019-05-30 12:40:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T12:40:00Z obsTimeLocal: 2019-05-30 22:40:00
> epoch: 1559220300 date_epoch_local: 2019-05-30 22:45:00 utcepoch: 
> 1559219300 date_epoch_utc: 2019-05-30 12:45:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T12:45:00Z obsTimeLocal: 2019-05-30 22:45:00
> epoch: 1559220600 date_epoch_local: 2019-05-30 22:50:00 utcepoch: 
> 1559219600 date_epoch_utc: 2019-05-30 12:50:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T12:50:00Z obsTimeLocal: 2019-05-30 22:50:00
> Number of WU records:  275
>
>
> Thu May 30 23:58:02 2019 UTC+10
> epoch: 1559223900 date_epoch_local: 2019-05-30 23:45:00 utcepoch: 
> 1559222900 date_epoch_utc: 2019-05-30 13:45:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T13:45:00Z obsTimeLocal: 2019-05-30 23:45:00
> epoch: 1559224200 date_epoch_local: 2019-05-30 23:50:00 utcepoch: 
> 1559223200 date_epoch_utc: 2019-05-30 13:50:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T13:50:00Z obsTimeLocal: 2019-05-30 23:50:00
> Number of WU records:  287
>
>
> 10 hour period after midnight on May 31. Request is for May 30 data. 
> Replies are May 31 data.  Local Date is 1 day ahead of UTC date
>
> Fri May 31 00:02:02 2019 UTC+10
> Could not get Weather Underground data.
> Reason: No JSON object could be decoded
> Could not get Weather Underground data. Exiting. 
>
>
> Fri May 31 00:58:02 2019  UTC+10
> epoch: 1559227500 date_epoch_local: 2019-05-31 00:45:00 utcepoch: 
> 1559226500 date_epoch_utc: 2019-05-30 14:45:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T14:45:00Z obsTimeLocal: 2019-05-31 00:45:00
> epoch: 1559227800 date_epoch_local: 2019-05-31 00:50:00 utcepoch: 
> 1559226800 date_epoch_utc: 2019-05-30 14:50:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T14:50:00Z obsTimeLocal: 2019-05-31 00:50:00
> Number of WU records:  11
>
>
> Fri May 31 02:58:03 2019 UTC+10
> epoch: 1559234700 date_epoch_local: 2019-05-31 02:45:00 utcepoch: 
> 1559233700 date_epoch_utc: 2019-05-30 16:45:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T16:45:00Z obsTimeLocal: 2019-05-31 02:45:00
> epoch: 1559235000 date_epoch_local: 2019-05-31 02:50:00 utcepoch: 
> 1559234000 date_epoch_utc: 2019-05-30 16:50:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T16:50:00Z obsTimeLocal: 2019-05-31 02:50:00
> Number of WU records:  35
>
>
> Fri May 31 05:58:02 2019 UTC+10
> epoch: 1559245500 date_epoch_local: 2019-05-31 05:45:00 utcepoch: 
> 1559244500 date_epoch_utc: 2019-05-30 19:45:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T19:45:00Z obsTimeLocal: 2019-05-31 05:45:00
> epoch: 1559245800 date_epoch_local: 2019-05-31 05:50:00 utcepoch: 
> 1559244800 date_epoch_utc: 2019-05-30 19:50:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T19:50:00Z obsTimeLocal: 2019-05-31 05:50:00
> Number of WU records:  71
>
>
> Fri May 31 08:58:03 2019 UTC+10
> epoch: 1559256300 date_epoch_local: 2019-05-31 08:45:00 utcepoch: 
> 1559255300 date_epoch_utc: 2019-05-30 22:45:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-30T22:45:00Z obsTimeLocal: 2019-05-31 08:45:00
> epoch: 1559256600 

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Rod Yager]

I constructed a cron job which exectued  wunderdates --date=2019-05-30 
 twice every hour, once 2 minutes before the hour and once 2 minutes after 
the hour.

Consistent with my earlier post, between 00:00+10 and 09:59+10 the reply 
from the api had data which belonged to 2019-05-31 - the day after the 
requested date. Before midnight, and after 10am local time, the reply from 
the api had data which belonged to 2019-05-30 (as requested).

I've pasted below some representative samples of the last 3 lines of the 
wunderdates output.

Before midnight on May 30: Request is for May 30 data. Replies are May 30 
data  Local Date = UTC date.

Thu May 30 20:58:02 2019 UTC+10
epoch: 1559213100 date_epoch_local: 2019-05-30 20:45:00 utcepoch: 
1559212100 date_epoch_utc: 2019-05-30 10:45:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T10:45:00Z obsTimeLocal: 2019-05-30 20:45:00
epoch: 1559213400 date_epoch_local: 2019-05-30 20:50:00 utcepoch: 
1559212400 date_epoch_utc: 2019-05-30 10:50:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T10:50:00Z obsTimeLocal: 2019-05-30 20:50:00
Number of WU records:  251


Thu May 30 22:58:03 2019 UTC+10
epoch: 155922 date_epoch_local: 2019-05-30 22:40:00 utcepoch: 
1559219000 date_epoch_utc: 2019-05-30 12:40:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T12:40:00Z obsTimeLocal: 2019-05-30 22:40:00
epoch: 1559220300 date_epoch_local: 2019-05-30 22:45:00 utcepoch: 
1559219300 date_epoch_utc: 2019-05-30 12:45:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T12:45:00Z obsTimeLocal: 2019-05-30 22:45:00
epoch: 1559220600 date_epoch_local: 2019-05-30 22:50:00 utcepoch: 
1559219600 date_epoch_utc: 2019-05-30 12:50:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T12:50:00Z obsTimeLocal: 2019-05-30 22:50:00
Number of WU records:  275


Thu May 30 23:58:02 2019 UTC+10
epoch: 1559223900 date_epoch_local: 2019-05-30 23:45:00 utcepoch: 
1559222900 date_epoch_utc: 2019-05-30 13:45:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T13:45:00Z obsTimeLocal: 2019-05-30 23:45:00
epoch: 1559224200 date_epoch_local: 2019-05-30 23:50:00 utcepoch: 
1559223200 date_epoch_utc: 2019-05-30 13:50:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T13:50:00Z obsTimeLocal: 2019-05-30 23:50:00
Number of WU records:  287


10 hour period after midnight on May 31. Request is for May 30 data. 
Replies are May 31 data.  Local Date is 1 day ahead of UTC date

Fri May 31 00:02:02 2019 UTC+10
Could not get Weather Underground data.
Reason: No JSON object could be decoded
Could not get Weather Underground data. Exiting. 


Fri May 31 00:58:02 2019  UTC+10
epoch: 1559227500 date_epoch_local: 2019-05-31 00:45:00 utcepoch: 
1559226500 date_epoch_utc: 2019-05-30 14:45:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T14:45:00Z obsTimeLocal: 2019-05-31 00:45:00
epoch: 1559227800 date_epoch_local: 2019-05-31 00:50:00 utcepoch: 
1559226800 date_epoch_utc: 2019-05-30 14:50:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T14:50:00Z obsTimeLocal: 2019-05-31 00:50:00
Number of WU records:  11


Fri May 31 02:58:03 2019 UTC+10
epoch: 1559234700 date_epoch_local: 2019-05-31 02:45:00 utcepoch: 
1559233700 date_epoch_utc: 2019-05-30 16:45:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T16:45:00Z obsTimeLocal: 2019-05-31 02:45:00
epoch: 1559235000 date_epoch_local: 2019-05-31 02:50:00 utcepoch: 
1559234000 date_epoch_utc: 2019-05-30 16:50:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T16:50:00Z obsTimeLocal: 2019-05-31 02:50:00
Number of WU records:  35


Fri May 31 05:58:02 2019 UTC+10
epoch: 1559245500 date_epoch_local: 2019-05-31 05:45:00 utcepoch: 
1559244500 date_epoch_utc: 2019-05-30 19:45:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T19:45:00Z obsTimeLocal: 2019-05-31 05:45:00
epoch: 1559245800 date_epoch_local: 2019-05-31 05:50:00 utcepoch: 
1559244800 date_epoch_utc: 2019-05-30 19:50:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T19:50:00Z obsTimeLocal: 2019-05-31 05:50:00
Number of WU records:  71


Fri May 31 08:58:03 2019 UTC+10
epoch: 1559256300 date_epoch_local: 2019-05-31 08:45:00 utcepoch: 
1559255300 date_epoch_utc: 2019-05-30 22:45:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T22:45:00Z obsTimeLocal: 2019-05-31 08:45:00
epoch: 1559256600 date_epoch_local: 2019-05-31 08:50:00 utcepoch: 
1559255600 date_epoch_utc: 2019-05-30 22:50:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T22:50:00Z obsTimeLocal: 2019-05-31 08:50:00
Number of WU records:  107


Fri May 31 09:58:03 2019 UTC+10
epoch: 1559259900 date_epoch_local: 2019-05-31 09:45:00 utcepoch: 
1559258900 date_epoch_utc: 2019-05-30 23:45:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T23:45:00Z obsTimeLocal: 2019-05-31 09:45:00
epoch: 1559260200 date_epoch_local: 2019-05-31 09:50:00 utcepoch: 
1559259200 date_epoch_utc: 2019-05-30 23:50:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T23:50:00Z obsTimeLocal: 2019-05-31 09:50:00
Number of WU records:  119

After 10AM on May 31: Request is for May 30 data. Repli

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Rod Yager]

Have downloaded the wunderdates. 

I am in the UTC+10 timezone.

So far, my experience is that when the current time is between 00:00+10 and 
09:59+10  (14:00UTC to 23:59 UTC) (so the local date is one day in advance 
of the UTC date), the API fetches data for the local-time day immediately 
following the date specified in the URL.  ie a request with date=20190528 
returns the data observations recorded on 20190529 local time.

At all other times (so far) the api returns the data for the local time day 
specified in the request URL (as expected).

>From what I understand, you are in the UTC-4 time zone and your experience 
is that when the current time is between 20:00-4 and 23:59-4 (00:00UTC to 
03:59UTC) (so the local date is one day behind the UTC date), the API 
fetches data for the local-time date immediately before the date specified 
in the URL. ie a request with date=20190529 returns the data for 
observations recorded on 20190527 local time. 

At all other times, I gather the api returns the data for the local time 
day specified in the request URL (as expected).

I'll conduct further tests, and will certainly report anything that doesn't 
fit my hypothesis that the unwanted behaviour reflects the difference 
between the current local time date and the current UTC date. But its 
difficult to see how there could be more to it than this. (Famous last 
words).

It's now 20:14+10 on 20190530  (10:14UTC on 20190503) The last three lines 
of  produced by ./wunderdates  are (as expected)

epoch: 1559210400 date_epoch_local: 2019-05-30 20:00:00 utcepoch: 
1559209400 date_epoch_utc: 2019-05-30 10:00:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T10:00:00Z obsTimeLocal: 2019-05-30 20:00:00

epoch: 1559210700 date_epoch_local: 2019-05-30 20:05:00 utcepoch: 
1559209700 date_epoch_utc: 2019-05-30 10:05:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-30T10:05:00Z obsTimeLocal: 2019-05-30 20:05:00

Number of WU records:  242


Rod

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/59d22f5e-051c-459d-bc38-b02c087a21c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Rod Yager]

Further to this, it has now rolled past 10am here, so the local date is now 
the same as the UTC date. (ie. local time May 30 2019 10:40 AM = May 30 
2019 00:40 AM UTC).

Now I get:

./wunderfixer --verbose --date=2019-05-29 --epsilon=125

Using configuration file /home/weewx/weewx.conf.

Using database binding 'wx_binding', which is bound to database 
'archive_mysql'

Weather Underground Station:   x

Date to check: 2019-05-29

Number of archive records: 288

Number of WU records:  288

Number of missing records: 0

[root@moses bin]# ./wunderfixer --verbose --date=2019-05-30 --epsilon=125

Using configuration file /home/weewx/weewx.conf.

Using database binding 'wx_binding', which is bound to database 
'archive_mysql'

Weather Underground Station:   xx

Date to check: 2019-05-30

Number of archive records: 128

Number of WU records:  127

Number of missing records: 1

This means that WU is now actually providing the records for the date 
requested, rather than the day after the requested date. 
So it seems that what is happening is that WU is determining whether or not 
the current date at the station location is the same as the UTC date.
If it is, it returns the data for the date as in the request. But if the 
local date is different, it makes an (unwanted) adjustment for the date 
difference.

Rod 


On Thursday, May 30, 2019 at 9:29:16 AM UTC+10, Leon Shaner wrote:
>
> Hi, Rod,
>
> Yes and thanks for adding yet another confirmation of the issue.  =D
>
> I can show that if I do the query within X hours of my offset of UTC, what 
> actually happens is they report 288 records from the day PRIOR to the one I 
> am asking about.
> For example, I ask for 20190528 and they give me records for 20190527, so 
> *that* is why wunderfixer "thinks" it needs to re-upload everything.
>
> I am in contact with IBM about it and have shown them irrefutable proof of 
> the issue.
> They didn't respond back yet, which I expect is because the proof was 
> irrefutable.  Ha!  ;-)
>
> I expect that they're investigating and would rather respond from a 
> position of understanding, or with any luck maybe even a quick fix.  =D
>
> I meant to follow-up with IBM again this morning, but got waylaid, so I'll 
> do that now.
> Thanks again, and for the reminder.  =D 
>
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>
> On May 29, 2019, at 6:14 PM, Rod Yager > 
> wrote:
>
> There is definitely a time zone issue. I am in the Sydney Australia 
> timezone (UTC +10 hours).
>
> It is currently 8am local time on May 30, 2019.   (10pm May 29, 2019 UTC)
>
> If I execute
>
> ./wunderfixer --verbose --date=2019-05-29 --epsilon=125
>
>
> I get
>
>
> Using configuration file /home/weewx/weewx.conf.
>
> Using database binding 'wx_binding', which is bound to database 
> 'archive_mysql'
>
> Weather Underground Station:   xxx
>
> Date to check: 2019-05-29
>
> Number of archive records: 288
>
> Number of WU records:  97
>
> Number of missing records: 288
>
>
> Now WU actually has 288 records for 2019-05-29.
>
> But it only has 97 records for 2019-05-30.
>
>
> So it is clear that wunderfixer is downloading the record data for 
> 2019-05-30 from WeatherUnderground and trying to match them with the local 
> records for 2019-30-29.
>
> Of course, they all mismatch, and so wunderfixer concludes that it must 
> upload all the data for 2019-05-29.
>
>
> Hope this narrows down the search for a solution.
>
>
> Rod
>
>
>
> On Monday, May 27, 2019 at 9:35:25 PM UTC+10, Leon Shaner wrote:
>>
>>
>> On May 27, 2019, at 12:12 AM, gjr80  wrote:
>>
>> On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
>>>
>>> [snip]
>>>
>> If you can see any shorter paths to a more reliable outcome than I have 
>>> achieved so far, then you know know know I will be very grateful.  =D
>>>
>>
>> I am not sure what local/UTC issue you refer to. When I do a 
>> api.weather.com/v2/pws/history query on a station to the east of 
>> Greenwich I am returned all records for the date specified (eg 20190525 
>> gives me all records for 25 May 2019), each record contains an epoch 
>> timestamp which is correct and consistent with 25 May 2019. Everything is 
>> as I would expect. However, when I perform the same query on a station to 
>> the west of Greenwich I am returned records for the day before the date 
>> specified (ie 20190525 gives me all records for 24 May 2019 not 25 May 
>> 2019), again each record contains an epoch timestamp but the timestamp is 
>> for the previous day Ie 24 May 2019. I have checked a number of data 
>> records in the stations history table and WU is definitely returning the 
>> midnight to midnight data for the day before. I have confirmed this 
>> behaviour with a number of stations both east and west of Greenwich.
>>
>> I don't think there is a local/UTC time issue, I think WU is having some 
>> implementation issues and for st

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Hi, Rod,

Yes and thanks for adding yet another confirmation of the issue.  =D

I can show that if I do the query within X hours of my offset of UTC, what 
actually happens is they report 288 records from the day PRIOR to the one I am 
asking about.
For example, I ask for 20190528 and they give me records for 20190527, so 
*that* is why wunderfixer "thinks" it needs to re-upload everything.

I am in contact with IBM about it and have shown them irrefutable proof of the 
issue.
They didn't respond back yet, which I expect is because the proof was 
irrefutable.  Ha!  ;-)

I expect that they're investigating and would rather respond from a position of 
understanding, or with any luck maybe even a quick fix.  =D

I meant to follow-up with IBM again this morning, but got waylaid, so I'll do 
that now.
Thanks again, and for the reminder.  =D 

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 29, 2019, at 6:14 PM, Rod Yager  wrote:
> 
> There is definitely a time zone issue. I am in the Sydney Australia timezone 
> (UTC +10 hours).
> 
> It is currently 8am local time on May 30, 2019.   (10pm May 29, 2019 UTC)
> 
> If I execute
> 
> ./wunderfixer --verbose --date=2019-05-29 --epsilon=125
> 
> 
> 
> I get
> 
> 
> 
> Using configuration file /home/weewx/weewx.conf.
> 
> Using database binding 'wx_binding', which is bound to database 
> 'archive_mysql'
> 
> Weather Underground Station:   xxx
> 
> Date to check: 2019-05-29
> 
> Number of archive records: 288
> 
> Number of WU records:  97
> 
> Number of missing records: 288
> 
> 
> 
> Now WU actually has 288 records for 2019-05-29.
> 
> But it only has 97 records for 2019-05-30.
> 
> 
> 
> So it is clear that wunderfixer is downloading the record data for 2019-05-30 
> from WeatherUnderground and trying to match them with the local records for 
> 2019-30-29.
> 
> Of course, they all mismatch, and so wunderfixer concludes that it must 
> upload all the data for 2019-05-29.
> 
> 
> 
> Hope this narrows down the search for a solution.
> 
> 
> 
> Rod
> 
> 
> 
> 
>> On Monday, May 27, 2019 at 9:35:25 PM UTC+10, Leon Shaner wrote:
>> 
>>> On May 27, 2019, at 12:12 AM, gjr80  wrote:
>>> 
 On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
 [snip]
 If you can see any shorter paths to a more reliable outcome than I have 
 achieved so far, then you know know know I will be very grateful.  =D
>>> 
>>> I am not sure what local/UTC issue you refer to. When I do a 
>>> api.weather.com/v2/pws/history query on a station to the east of Greenwich 
>>> I am returned all records for the date specified (eg 20190525 gives me all 
>>> records for 25 May 2019), each record contains an epoch timestamp which is 
>>> correct and consistent with 25 May 2019. Everything is as I would expect. 
>>> However, when I perform the same query on a station to the west of 
>>> Greenwich I am returned records for the day before the date specified (ie 
>>> 20190525 gives me all records for 24 May 2019 not 25 May 2019), again each 
>>> record contains an epoch timestamp but the timestamp is for the previous 
>>> day Ie 24 May 2019. I have checked a number of data records in the stations 
>>> history table and WU is definitely returning the midnight to midnight data 
>>> for the day before. I have confirmed this behaviour with a number of 
>>> stations both east and west of Greenwich.
>>> 
>>> I don't think there is a local/UTC time issue, I think WU is having some 
>>> implementation issues and for stations west of Greenwich they are returning 
>>> the wrong day of data.
>> 
>> Thanks, Gary!  This was all very helpful.
>> In addition to what you've described across the east vs west of GMT, I get 
>> similar behavior if I am within X hours of my local UTC offset when querying 
>> my own station.
>> Last night as soon as localtime rolled over midnight, the queries for the 
>> previous day were correct.
>> 
>> --Leon
>> 
>>> 
>>> Gary
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/683a28af-35e4-474e-95a0-f684b9926af0%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/FD8C60DC-D403-4411-8A6D-B088AFFC90FC%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Rod Yager]

There is definitely a time zone issue. I am in the Sydney Australia 
timezone (UTC +10 hours).

It is currently 8am local time on May 30, 2019.   (10pm May 29, 2019 UTC)

If I execute

./wunderfixer --verbose --date=2019-05-29 --epsilon=125


I get


Using configuration file /home/weewx/weewx.conf.

Using database binding 'wx_binding', which is bound to database 
'archive_mysql'

Weather Underground Station:   xxx

Date to check: 2019-05-29

Number of archive records: 288

Number of WU records:  97

Number of missing records: 288


Now WU actually has 288 records for 2019-05-29.

But it only has 97 records for 2019-05-30.


So it is clear that wunderfixer is downloading the record data for 
2019-05-30 from WeatherUnderground and trying to match them with the local 
records for 2019-30-29.

Of course, they all mismatch, and so wunderfixer concludes that it must 
upload all the data for 2019-05-29.


Hope this narrows down the search for a solution.


Rod



On Monday, May 27, 2019 at 9:35:25 PM UTC+10, Leon Shaner wrote:
>
>
> On May 27, 2019, at 12:12 AM, gjr80 > 
> wrote:
>
> On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
>>
>> [snip]
>>
> If you can see any shorter paths to a more reliable outcome than I have 
>> achieved so far, then you know know know I will be very grateful.  =D
>>
>
> I am not sure what local/UTC issue you refer to. When I do a 
> api.weather.com/v2/pws/history query on a station to the east of 
> Greenwich I am returned all records for the date specified (eg 20190525 
> gives me all records for 25 May 2019), each record contains an epoch 
> timestamp which is correct and consistent with 25 May 2019. Everything is 
> as I would expect. However, when I perform the same query on a station to 
> the west of Greenwich I am returned records for the day before the date 
> specified (ie 20190525 gives me all records for 24 May 2019 not 25 May 
> 2019), again each record contains an epoch timestamp but the timestamp is 
> for the previous day Ie 24 May 2019. I have checked a number of data 
> records in the stations history table and WU is definitely returning the 
> midnight to midnight data for the day before. I have confirmed this 
> behaviour with a number of stations both east and west of Greenwich.
>
> I don't think there is a local/UTC time issue, I think WU is having some 
> implementation issues and for stations west of Greenwich they are returning 
> the wrong day of data.
>
>
> Thanks, Gary!  This was all very helpful.
> In addition to what you've described across the east vs west of GMT, I get 
> similar behavior if I am within X hours of my local UTC offset when 
> querying my own station.
> Last night as soon as localtime rolled over midnight, the queries for the 
> previous day were correct.
>
> --Leon
>
>
> Gary
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/683a28af-35e4-474e-95a0-f684b9926af0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

> On May 27, 2019, at 12:12 AM, gjr80  wrote:
> 
>> On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
>> [snip]
>> If you can see any shorter paths to a more reliable outcome than I have 
>> achieved so far, then you know know know I will be very grateful.  =D
> 
> I am not sure what local/UTC issue you refer to. When I do a 
> api.weather.com/v2/pws/history query on a station to the east of Greenwich I 
> am returned all records for the date specified (eg 20190525 gives me all 
> records for 25 May 2019), each record contains an epoch timestamp which is 
> correct and consistent with 25 May 2019. Everything is as I would expect. 
> However, when I perform the same query on a station to the west of Greenwich 
> I am returned records for the day before the date specified (ie 20190525 
> gives me all records for 24 May 2019 not 25 May 2019), again each record 
> contains an epoch timestamp but the timestamp is for the previous day Ie 24 
> May 2019. I have checked a number of data records in the stations history 
> table and WU is definitely returning the midnight to midnight data for the 
> day before. I have confirmed this behaviour with a number of stations both 
> east and west of Greenwich.
> 
> I don't think there is a local/UTC time issue, I think WU is having some 
> implementation issues and for stations west of Greenwich they are returning 
> the wrong day of data.

Thanks, Gary!  This was all very helpful.
In addition to what you've described across the east vs west of GMT, I get 
similar behavior if I am within X hours of my local UTC offset when querying my 
own station.
Last night as soon as localtime rolled over midnight, the queries for the 
previous day were correct.

--Leon

> 
> Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/7BBF146A-7408-44AD-8676-FDE99ED6532C%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Gary,

It's late, so I'll respond to the rest later, but...

The problem here is that if we compare our local every 1-minute records to WU's 
query that only shows every 5-minute records, then we'll keep re-uploading the 
the "missing" 1-minute records every time wunderfixer is called.  This amounts 
to pointless hits against WU's infrastructure, since pragmatically it's very 
clear they almost always drop all records not closely aligned to 5-minute 
"buckets."

I'm trying to get to the heart of the matter re: what wunderfixer was really 
designed to do, which is to make sure what we have locally is consistent with 
what WU will actually REPORT, and when there are gaps in what WU *should* 
report, and we have local data to fill in those gaps, then re-upload.  BUT to 
not gratuitously re-upload data that WU generally throws away (they just don't 
seem to care about storing data at < 5-minute intervals).

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 27, 2019, at 12:12 AM, gjr80  wrote:
> 
>> On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
>> Gary,
>> 
>> In practice, WU seems to discard data that is not close to their APPARENTly 
>> preferred 5-minute "normalization buckets."
>> 
>> I upload via rapidfire *and* regular loop on 1-minute intervals, and 
>> irregardless of same, the queries only ever show the records most closely 
>> aligned to their "preferred" 5-minute "buckets."
>> 
>> I would love to see them preserve the same interval that I upload, but 
>> either by design or omission or bug, they simply don't.  :-(
>> Could be the layers in-between are dropping data by code-exception.
>> Could be it's deliberate.
>> I'm not here to debug their code unless they want to start paying me a 
>> worthy salary. ;-)
>> 
>> Well...  MOST of the time the behavior I have described seems to be true.  
>> :-/
>> 
>> While WU normally SEEMS to only keeps records on 5-minute boundaries, I have 
>> seen where if wunderfixer is used persistently enough (say more than 10 
>> times spread out every 20 minutes across a total span of 200 minutes) then 
>> it will keep records that are more frequent than every 5-minutes.
>> 
>> Right now / especially lately, things are so sporadic with WU that I don't 
>> even want to spend cycles chasing what's really going on there.  I would 
>> rather just only have wunderfixer ignore most records except those nearest 
>> the 5-minute boundaries that WU seems to most care about.
>> 
> 
> Show Quoted Content
>> Gary,
>> 
>> In practice, WU seems to discard data that is not close to their APPARENTly 
>> preferred 5-minute "normalization buckets."
>> 
>> I upload via rapidfire *and* regular loop on 1-minute intervals, and 
>> irregardless of same, the queries only ever show the records most closely 
>> aligned to their "preferred" 5-minute "buckets."
>> 
>> I would love to see them preserve the same interval that I upload, but 
>> either by design or omission or bug, they simply don't.  :-(
>> Could be the layers in-between are dropping data by code-exception.
>> Could be it's deliberate.
>> I'm not here to debug their code unless they want to start paying me a 
>> worthy salary. ;-)
>> 
>> Well...  MOST of the time the behavior I have described seems to be true.  
>> :-/
>> 
>> While WU normally SEEMS to only keeps records on 5-minute boundaries, I have 
>> seen where if wunderfixer is used persistently enough (say more than 10 
>> times spread out every 20 minutes across a total span of 200 minutes) then 
>> it will keep records that are more frequent than every 5-minutes.
>> 
>> Right now / especially lately, things are so sporadic with WU that I don't 
>> even want to spend cycles chasing what's really going on there.  I would 
>> rather just only have wunderfixer ignore most records except those nearest 
>> the 5-minute boundaries that WU seems to most care about.
>> 
> I am not suggesting we/you/anyone should try to solve WU 
> issues/problems/whatever, merely making the point that WU is a blackbox, 
> nobody seems to really know what it does with the data that it is fed. If we 
> are going to have WU recreate/create 'missing' data then the logical approach 
> would be to feed it with the data it was orignally fed rather than trying to 
> guess what it wants.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/8B0BFD31-7D91-4A3D-BDED-8923E8AF0C04%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[gjr80]

On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
>
> Gary,
>
> In practice, WU seems to discard data that is not close to their 
> APPARENTly preferred 5-minute "normalization buckets."
>
> I upload via rapidfire *and* regular loop on 1-minute intervals, and 
> irregardless of same, the queries only ever show the records most closely 
> aligned to their "preferred" 5-minute "buckets."
>
> I would love to see them preserve the same interval that I upload, but 
> either by design or omission or bug, they simply don't.  :-(
> Could be the layers in-between are dropping data by code-exception.
> Could be it's deliberate.
> I'm not here to debug their code unless they want to start paying me a 
> worthy salary. ;-)
>
> Well...  MOST of the time the behavior I have described seems to be true. 
>  :-/
>
> While WU normally SEEMS to only keeps records on 5-minute boundaries, I 
> have seen where if wunderfixer is used persistently enough (say more than 
> 10 times spread out every 20 minutes across a total span of 200 minutes) 
> then it will keep records that are more frequent than every 5-minutes.
>
> Right now / especially lately, things are so sporadic with WU that I don't 
> even want to spend cycles chasing what's really going on there.  I would 
> rather just only have wunderfixer ignore most records except those nearest 
> the 5-minute boundaries that WU seems to most care about.
>
> I am not suggesting we/you/anyone should try to solve WU 
issues/problems/whatever, merely making the point that WU is a blackbox, 
nobody seems to really know what it does with the data that it is fed. If 
we are going to have WU recreate/create 'missing' data then the logical 
approach would be to feed it with the data it was orignally fed rather than 
trying to guess what it wants.
 

> I have just now solved the "align to 5-minute boundaries" exercise with:
>
> _gen_rows =  dbmanager.genSql("""SELECT dateTime FROM archive WHERE 
> dateTime>=? AND dateTime   (start_ts, end_ts))
>
> Example:
>
> $ /usr/share/weewx4/bin/wunderfixer_403 --epsilon=125 --date=2019-05-25 
> --test --verbose | head -30
> Using configuration file /usr/share/weewx4/weewx.conf.
> Using database binding 'wx_binding', which is bound to database 
> 'archive_sqlite'
> Weather Underground Station:   KMIDEARB5
> Date to check: 2019-05-25
> Number of archive records: 289
> Number of WU records:  260
> Number of missing records: 288
>
> Missing records:
> 2019-05-25 00:05:00 EDT (1558757100); 29.326";  61.3F;  76%; 0.0 mph; N/A 
> deg; 3.6 mph gust;  53.7F; 0.00" rain  ... skipped.
> 2019-05-25 00:10:00 EDT (1558757400); 29.326";  61.2F;  76%; 0.0 mph; N/A 
> deg; 2.0 mph gust;  53.5F; 0.00" rain  ... skipped.
> 2019-05-25 00:15:00 EDT (1558757700); 29.326";  61.2F;  76%; 1.3 mph;  93 
> deg; 3.1 mph gust;  53.5F; 0.00" rain  ... skipped.
> 2019-05-25 00:20:00 EDT (1558758000); 29.317";  61.1F;  76%; 1.3 mph;  93 
> deg; 2.5 mph gust;  53.5F; 0.00" rain  ... skipped.
> 2019-05-25 00:25:00 EDT (1558758300); 29.317";  61.0F;  77%; 1.3 mph; 121 
> deg; 4.3 mph gust;  53.7F; 0.00" rain  ... skipped.
> 2019-05-25 00:30:00 EDT (1558758600); 29.317";  61.0F;  76%; 1.3 mph; 121 
> deg; 3.6 mph gust;  53.4F; 0.00" rain  ... skipped.
> 2019-05-25 00:35:00 EDT (1558758900); 29.309";  61.0F;  77%; 1.8 mph; 112 
> deg; 4.0 mph gust;  53.7F; 0.00" rain  ... skipped.
> 2019-05-25 00:40:00 EDT (1558759200); 29.309";  60.9F;  76%; 1.8 mph; 112 
> deg; 3.1 mph gust;  53.3F; 0.00" rain  ... skipped.
> 2019-05-25 00:45:00 EDT (1558759500); 29.309";  60.7F;  77%; 1.8 mph; 116 
> deg; 5.8 mph gust;  53.3F; 0.00" rain  ... skipped.
> 2019-05-25 00:50:00 EDT (1558759800); 29.306";  60.6F;  77%; 1.8 mph; 116 
> deg; 5.1 mph gust;  53.4F; 0.00" rain  ... skipped.
> 2019-05-25 00:55:00 EDT (1558760100); 29.306";  60.6F;  77%; 2.5 mph; 113 
> deg; 2.9 mph gust;  53.4F; 0.00" rain  ... skipped.
> 2019-05-25 01:00:00 EDT (1558760400); 29.306";  60.5F;  77%; 2.5 mph; 113 
> deg; 2.5 mph gust;  53.3F; 0.00" rain  ... skipped.
> 2019-05-25 01:05:00 EDT (1558760700); 29.303";  60.5F;  77%; 1.3 mph; 120 
> deg; 3.6 mph gust;  53.2F; 0.00" rain  ... skipped.
> 2019-05-25 01:10:00 EDT (1558761000); 29.303";  60.4F;  78%; 1.3 mph; 120 
> deg; 1.3 mph gust;  53.5F; 0.00" rain  ... skipped.
> 2019-05-25 01:15:00 EDT (1558761300); 29.303";  60.4F;  78%; 1.3 mph; 110 
> deg; 4.7 mph gust;  53.5F; 0.00" rain  ... skipped.
> 2019-05-25 01:20:00 EDT (1558761600); 29.285";  60.4F;  78%; 1.3 mph; 110 
> deg; 5.1 mph gust;  53.5F; 0.00" rain  ... skipped.
> 2019-05-25 01:25:00 EDT (1558761900); 29.285";  60.3F;  78%; 2.0 mph; 102 
> deg; 3.1 mph gust;  53.4F; 0.00" rain  ... skipped.
> 2019-05-25 01:30:00 EDT (1558762200); 29.285";  60.3F;  78%; 2.0 mph; 102 
> deg; 5.4 mph gust;  53.4F; 0.00" rain  ... skipped.
> 2019-05-25 01:35:00 EDT (1558762500); 29.282";  60.2F;  78%; 2.5 mph; 123 
> deg; 2.5 mph gust;  53.3F; 0.00" rain  ... 

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Gary,

In practice, WU seems to discard data that is not close to their APPARENTly 
preferred 5-minute "normalization buckets."

I upload via rapidfire *and* regular loop on 1-minute intervals, and 
irregardless of same, the queries only ever show the records most closely 
aligned to their "preferred" 5-minute "buckets."

I would love to see them preserve the same interval that I upload, but either 
by design or omission or bug, they simply don't.  :-(
Could be the layers in-between are dropping data by code-exception.
Could be it's deliberate.
I'm not here to debug their code unless they want to start paying me a worthy 
salary. ;-)

Well...  MOST of the time the behavior I have described seems to be true.  :-/

While WU normally SEEMS to only keeps records on 5-minute boundaries, I have 
seen where if wunderfixer is used persistently enough (say more than 10 times 
spread out every 20 minutes across a total span of 200 minutes) then it will 
keep records that are more frequent than every 5-minutes.

Right now / especially lately, things are so sporadic with WU that I don't even 
want to spend cycles chasing what's really going on there.  I would rather just 
only have wunderfixer ignore most records except those nearest the 5-minute 
boundaries that WU seems to most care about.

I have just now solved the "align to 5-minute boundaries" exercise with:

_gen_rows =  dbmanager.genSql("""SELECT dateTime FROM archive WHERE dateTime>=? 
AND dateTime On May 26, 2019, at 10:59 PM, gjr80  wrote:
> 
>> On Saturday, 25 May 2019 22:54:25 UTC+10, Leon Shaner wrote:
>> 
>> There is no point re-uploading 00:00:00, then 00:01:00, then 00:02:00, 
>> because WU is only going to keep 00:00:00, and then later it will keep 
>> whatever is closest to 00:05:00.   That's been the behavior of wunderfixer 
>> vs. WU for as long as I've been working with it, however.  :-/
> 
> Maybe, maybe not. The problem is WU acts as a black box that accepts your 
> data at (more or less) whatever interval you choose WU then does who knows 
> what with the data and (usually) 5 minute interval records appear. You could 
> argue that re-uploading of data should be done in the same manner as it was 
> originally uploaded.
> 
> Gary
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/c3476bc2-674f-4928-bf58-55f2150159b2%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/309313C3-2AD2-43B4-941C-7BBD90DB35C8%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[gjr80]

On Saturday, 25 May 2019 22:54:25 UTC+10, Leon Shaner wrote:
>
>
> There is no point re-uploading 00:00:00, then 00:01:00, then 00:02:00, 
> because WU is only going to keep 00:00:00, and then later it will keep 
> whatever is closest to 00:05:00.   That's been the behavior of wunderfixer 
> vs. WU for as long as I've been working with it, however.  :-/
>

Maybe, maybe not. The problem is WU acts as a black box that accepts your 
data at (more or less) whatever interval you choose WU then does who knows 
what with the data and (usually) 5 minute interval records appear. You 
could argue that re-uploading of data should be done in the same manner as 
it was originally uploaded.

Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/c3476bc2-674f-4928-bf58-55f2150159b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Tested on 4.x (python 2 and 3) and 3.9.x. (python 2).

Ultimately went with apikey instead of apiKey in weewx.conf, which was to be 
more consistent with other parameter names.

If you are willing to pass "--apikey=0123456789012345678901" on wunderfixer 
command-line, then you only need wunderfixer.

Unrelated to these changes, FYI you probably also want --epsilon=125 because of 
silly WU "rounding" errors on fractional minutes.

If you wish to put "apikey=0123456789012345678901" in the WU section of 
weewx.conf, then you also need restx.py.

API Key is generated in the "Member Settings" section of wunderground.com ("My 
Profile" > "Member Settings" > "API Keys").

3.9.x:
https://github.com/UberEclectic/weewx/blob/master/bin/wunderfixer

https://github.com/UberEclectic/weewx/blob/master/bin/weewx/restx.py

4.x:
https://github.com/UberEclectic/weewx/blob/development/bin/wunderfixer

https://github.com/UberEclectic/weewx/blob/development/bin/weewx/restx.py

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 25, 2019, at 8:54 AM, Leon Shaner  wrote:
> 
> So this is where I am...
> These all seem reasonable (but then scroll for the problem which starts with 
> yesterday's date):
> 
> pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 
> --date=2019-05-21 --test --verbose | more
> Using configuration file /usr/share/weewx4/weewx.conf.
> Using database binding 'wx_binding', which is bound to database 
> 'archive_sqlite'
> Weather Underground Station:   KMIDEARB5
> Date to check: 2019-05-21
> Number of archive records: 1440
> Number of WU records:  288
> Number of missing records: 4
> 
> Missing records:
> 2019-05-21 00:00:00 EDT (1558411200); 29.379";  49.8F;  78%; 0.0 mph; N/A 
> deg; 0.0 mph gust;  43.2F; 0.00" rain  ... skipped.
> 2019-05-21 00:01:00 EDT (1558411260); 29.379";  49.7F;  78%; 0.0 mph; N/A 
> deg; 0.0 mph gust;  43.2F; 0.00" rain  ... skipped.
> 2019-05-21 00:02:00 EDT (1558411320); 29.379";  49.6F;  78%; 0.0 mph; N/A 
> deg; 0.0 mph gust;  43.1F; 0.00" rain  ... skipped.
> 2019-05-21 22:42:00 EDT (1558492920); 29.406";  55.4F;  67%; 1.3 mph;  68 
> deg; 0.0 mph gust;  44.6F; 0.00" rain  ... skipped.
> pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 
> --date=2019-05-22 --test --verbose | more
> Using configuration file /usr/share/weewx4/weewx.conf.
> Using database binding 'wx_binding', which is bound to database 
> 'archive_sqlite'
> Weather Underground Station:   KMIDEARB5
> Date to check: 2019-05-22
> Number of archive records: 1440
> Number of WU records:  288
> Number of missing records: 3
> 
> Missing records:
> 2019-05-22 00:00:00 EDT (1558497600); 29.406";  54.0F;  65%; 2.0 mph;  76 
> deg; 3.6 mph gust;  42.5F; 0.00" rain  ... skipped.
> 2019-05-22 00:01:00 EDT (1558497660); 29.406";  54.0F;  65%; 2.3 mph;  64 
> deg; 2.5 mph gust;  42.5F; 0.00" rain  ... skipped.
> 2019-05-22 00:02:00 EDT (1558497720); 29.406";  54.0F;  65%; 2.5 mph;  61 
> deg; 2.9 mph gust;  42.5F; 0.00" rain  ... skipped.
> pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 
> --date=2019-05-23 --test --verbose | more
> Using configuration file /usr/share/weewx4/weewx.conf.
> Using database binding 'wx_binding', which is bound to database 
> 'archive_sqlite'
> Weather Underground Station:   KMIDEARB5
> Date to check: 2019-05-23
> Number of archive records: 1438
> Number of WU records:  288
> Number of missing records: 4
> 
> Missing records:
> 2019-05-23 00:00:00 EDT (1558584000); 29.244";  59.9F;  90%; 1.3 mph; 158 
> deg; 1.8 mph gust;  56.9F; 0.00" rain  ... skipped.
> 2019-05-23 00:01:00 EDT (1558584060); 29.244";  59.8F;  90%; 1.3 mph; 159 
> deg; 1.8 mph gust;  56.8F; 0.00" rain  ... skipped.
> 2019-05-23 00:02:00 EDT (1558584120); 29.244";  59.7F;  90%; 1.3 mph; 160 
> deg; 1.3 mph gust;  56.8F; 0.00" rain  ... skipped.
> 2019-05-23 14:52:00 EDT (1558637520); 29.161";  83.8F;  46%; 3.6 mph; 248 
> deg;12.3 mph gust;  60.8F; 0.00" rain  ... skipped.
> 
> 
> Using configuration file /usr/share/weewx4/weewx.conf.
> Using database binding 'wx_binding', which is bound to database 
> 'archive_sqlite'
> Weather Underground Station:   KMIDEARB5
> Date to check: 2019-05-24
> Number of archive records: 1436
> Number of WU records:  260
> Number of missing records: 149
> 
> Missing records:
> 2019-05-24 00:00:00 EDT (1558670400); 29.320";  63.0F;  70%; 0.0 mph; N/A 
> deg; 0.0 mph gust;  52.9F; 0.00" rain  ... skipped.
> 2019-05-24 00:01:00 EDT (1558670460); 29.320";  63.0F;  70%; 0.0 mph; N/A 
> deg; 0.0 mph gust;  52.9F; 0.00" rain  ... skipped.
> 2019-05-24 00:02:00 EDT (1558670520); 29.320";  63.0F;  70%; 0.0 mph; N/A 
> deg; 0.0 mph gust;  52.9F; 0.00" rain  ... skipped.
> 2019-05-24 18:22:00 EDT (1558736520); 29.359";  67.5F;  68%; 2.5 mph;  99 
> deg; 5.1 mph gust;  56.5F; 0.00" rain  ... skipped.
> 2019-05-24 18:40:00 EDT (15587376

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

So this is where I am...
These all seem reasonable (but then scroll for the problem which starts with 
yesterday's date):

pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 --date=2019-05-21 
--test --verbose | more
Using configuration file /usr/share/weewx4/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check: 2019-05-21
Number of archive records: 1440
Number of WU records:  288
Number of missing records: 4

Missing records:
2019-05-21 00:00:00 EDT (1558411200); 29.379";  49.8F;  78%; 0.0 mph; N/A deg; 
0.0 mph gust;  43.2F; 0.00" rain  ... skipped.
2019-05-21 00:01:00 EDT (1558411260); 29.379";  49.7F;  78%; 0.0 mph; N/A deg; 
0.0 mph gust;  43.2F; 0.00" rain  ... skipped.
2019-05-21 00:02:00 EDT (1558411320); 29.379";  49.6F;  78%; 0.0 mph; N/A deg; 
0.0 mph gust;  43.1F; 0.00" rain  ... skipped.
2019-05-21 22:42:00 EDT (1558492920); 29.406";  55.4F;  67%; 1.3 mph;  68 deg; 
0.0 mph gust;  44.6F; 0.00" rain  ... skipped.
pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 --date=2019-05-22 
--test --verbose | more
Using configuration file /usr/share/weewx4/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check: 2019-05-22
Number of archive records: 1440
Number of WU records:  288
Number of missing records: 3

Missing records:
2019-05-22 00:00:00 EDT (1558497600); 29.406";  54.0F;  65%; 2.0 mph;  76 deg; 
3.6 mph gust;  42.5F; 0.00" rain  ... skipped.
2019-05-22 00:01:00 EDT (1558497660); 29.406";  54.0F;  65%; 2.3 mph;  64 deg; 
2.5 mph gust;  42.5F; 0.00" rain  ... skipped.
2019-05-22 00:02:00 EDT (1558497720); 29.406";  54.0F;  65%; 2.5 mph;  61 deg; 
2.9 mph gust;  42.5F; 0.00" rain  ... skipped.
pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 --date=2019-05-23 
--test --verbose | more
Using configuration file /usr/share/weewx4/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check: 2019-05-23
Number of archive records: 1438
Number of WU records:  288
Number of missing records: 4

Missing records:
2019-05-23 00:00:00 EDT (1558584000); 29.244";  59.9F;  90%; 1.3 mph; 158 deg; 
1.8 mph gust;  56.9F; 0.00" rain  ... skipped.
2019-05-23 00:01:00 EDT (1558584060); 29.244";  59.8F;  90%; 1.3 mph; 159 deg; 
1.8 mph gust;  56.8F; 0.00" rain  ... skipped.
2019-05-23 00:02:00 EDT (1558584120); 29.244";  59.7F;  90%; 1.3 mph; 160 deg; 
1.3 mph gust;  56.8F; 0.00" rain  ... skipped.
2019-05-23 14:52:00 EDT (1558637520); 29.161";  83.8F;  46%; 3.6 mph; 248 
deg;12.3 mph gust;  60.8F; 0.00" rain  ... skipped.


Using configuration file /usr/share/weewx4/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check: 2019-05-24
Number of archive records: 1436
Number of WU records:  260
Number of missing records: 149

Missing records:
2019-05-24 00:00:00 EDT (1558670400); 29.320";  63.0F;  70%; 0.0 mph; N/A deg; 
0.0 mph gust;  52.9F; 0.00" rain  ... skipped.
2019-05-24 00:01:00 EDT (1558670460); 29.320";  63.0F;  70%; 0.0 mph; N/A deg; 
0.0 mph gust;  52.9F; 0.00" rain  ... skipped.
2019-05-24 00:02:00 EDT (1558670520); 29.320";  63.0F;  70%; 0.0 mph; N/A deg; 
0.0 mph gust;  52.9F; 0.00" rain  ... skipped.
2019-05-24 18:22:00 EDT (1558736520); 29.359";  67.5F;  68%; 2.5 mph;  99 deg; 
5.1 mph gust;  56.5F; 0.00" rain  ... skipped.
2019-05-24 18:40:00 EDT (1558737600); 29.353";  67.3F;  68%; 2.0 mph; 118 deg; 
4.0 mph gust;  56.3F; 0.00" rain  ... skipped.
2019-05-24 18:41:00 EDT (1558737660); 29.353";  67.3F;  68%; 2.0 mph; 118 deg; 
3.6 mph gust;  56.3F; 0.00" rain  ... skipped.
...

When forming the query URL, I suspect I need to convert the local date to the 
UTC date, since I was within 4 hours of UTC midnight.  The docs will probably 
confirm it.

The thing is, late last night (US/Eastern), the number was over 1300.
If it's the UTC vs. EDT date in the URL, I would expect the records to line up 
when querying what is querying what is now yesterday.

And for today, so far, seems reasonable.

pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 --date=2019-05-25 
--test --verbose | more
Using configuration file /usr/share/weewx4/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check: 2019-05-25
Number of archive records: 526
Number of WU records:  106
Number of missing records: 3

Missing records:
2019-05-25 00:00:00 EDT (1558756800); 29.329";  61.4F;  75%; 0.0 mph; N/A deg; 
0.0 mph gust;  53.4F; 0.00" rain  ... skipped.
2019-05-25 00:01:00 EDT (1558756860); 29.32

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Hey, Gary,

I figured this out last night with a cue from Tom.  But it requires modifying 
two different classes in restx.py, similar to the way you mentioned and I am a 
little uncomfortable with it, since I don't know all the different touch points 
/ ramifications for other code.

Also, something strange started happening toward the end last night and the 
list of missing records grew to pretty much 100%.
I double checked and the new query is still pulling back the expected ~288 
records (every 5 mins, is 12 an hour, so 288 in a day), but I now need to 
scrutinize the "epoch" values, because it's  almost as if IBM changed them by 
some offset in the middle of the day, so now none of them "line up" with my 
local data.

I am heading out of town, so I will have to pick this up on Monday.

Thanks for the help! =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On May 25, 2019, at 1:24 AM, gjr80  wrote:
> 
> No, all get_site_dict does is check that the listed config options exist and 
> are not the install defaults. The real issue is that _ambient_dict is used to 
> pass keyword arguments to class AmbientThread init, apiKey is not a parameter 
> in the init signature so hence the error. One approach will be to simply add 
> an apiKey=None parameter to the AmbientClass init signature, it won't be used 
> but who knows where WU may end up, maybe it could be thought of as a bit of 
> future proofing/planning. Another approach would be to have a separate 
> [Wunderfixer] stanza in weewx.conf, seems a waste to me and it makes sense to 
> have the WU parameters together in the one place. Maybe leave it as a command 
> line parameter only for wunderfixer, to me that seems almost the same as 
> having a separate [Wunderfixer] stanza. Maybe introduce whatever approach in 
> 4.0 and leave it as a wunderfixer command line parameter for the (expected) 
> 3.9.2 release. Probably one for guidance from Tom.
> 
> Gary
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/68bfe21d-d0a2-4c55-9e71-2d65a69eea03%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAA23789-2FE9-4DCA-A2EC-386451439DC4%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[gjr80]

No, all get_site_dict does is check that the listed config options exist and 
are not the install defaults. The real issue is that _ambient_dict is used to 
pass keyword arguments to class AmbientThread init, apiKey is not a parameter 
in the init signature so hence the error. One approach will be to simply add an 
apiKey=None parameter to the AmbientClass init signature, it won't be used but 
who knows where WU may end up, maybe it could be thought of as a bit of future 
proofing/planning. Another approach would be to have a separate [Wunderfixer] 
stanza in weewx.conf, seems a waste to me and it makes sense to have the WU 
parameters together in the one place. Maybe leave it as a command line 
parameter only for wunderfixer, to me that seems almost the same as having a 
separate [Wunderfixer] stanza. Maybe introduce whatever approach in 4.0 and 
leave it as a wunderfixer command line parameter for the (expected) 3.9.2 
release. Probably one for guidance from Tom.

Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/68bfe21d-d0a2-4c55-9e71-2d65a69eea03%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Hey, Gary,

Below is the error on startup, simply because I added the "apiKey = blah" right 
after the "password = oog" in the [[Wunderground]] section.

This led me to believe that I need to somehow "extend" the ambient_dict, but 
for the life of me I am not seeing how to do that.  I looked every where for 
station and password in the restx.py and weecfg/__init__.py, and so far I am 
absolutely stumped.

Of course I appended to what seemed like an obvious place in restx.py, but 
nope.  No good:

_ambient_dict = get_site_dict(
config_dict, 'Wunderground', 'station', 'password', 'apiKey')
if _ambient_dict is None:
return

Same exact error without or with that added.  :-(

Any insights very much appreciated.  =D

May 24 19:59:03 nixie weewx[17779]: restx: WU essentials: {}
May 24 19:59:03 nixie weewx[17779]: engine: Caught unrecoverable exception in 
engine:
May 24 19:59:03 nixie weewx[17779]:   __init__() got an unexpected 
keyword argument 'apiKey'
May 24 19:59:03 nixie weewx[17779]:   : Traceback (most recent call 
last):
May 24 19:59:03 nixie weewx[17779]:   :   File 
"/usr/share/weewx4/bin/weewx/engine.py", line 892, in main
May 24 19:59:03 nixie weewx[17779]:   : engine = 
engine_class(config_dict)
May 24 19:59:03 nixie weewx[17779]:   :   File 
"/usr/share/weewx4/bin/weewx/engine.py", line 79, in __init__
May 24 19:59:03 nixie weewx[17779]:   : 
self.loadServices(config_dict)
May 24 19:59:03 nixie weewx[17779]:   :   File 
"/usr/share/weewx4/bin/weewx/engine.py", line 143, in loadServices
May 24 19:59:03 nixie weewx[17779]:   : 
self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))
May 24 19:59:03 nixie weewx[17779]:   :   File 
"/usr/share/weewx4/bin/weewx/restx.py", line 607, in __init__
May 24 19:59:03 nixie weewx[17779]:   : **_ambient_dict)
May 24 19:59:03 nixie weewx[17779]:   : TypeError: __init__() got an 
unexpected keyword argument 'apiKey'
May 24 19:59:03 nixie weewx[17779]:   Exiting.
May 24 19:59:03 nixie systemd[1]: weewx.service: Main process exited, 
code=exited, status=1/FAILURE
May 24 19:59:03 nixie systemd[1]: weewx.service: Unit entered failed state.
May 24 19:59:03 nixie systemd[1]: weewx.service: Failed with result 'exit-code'.


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 24, 2019, at 7:29 PM, gjr80  wrote:
> 
> What does the error trace show? WeeWX shouldn't be concerned about extra 
> config options. We don't need to re-invent wheels, forecast extension (among 
> others) already uses a WU API key, not appropriate to use that config option 
> here but using a consistent config option naming convention makes it easier 
> for users.
> 
> Gary
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/b069c856-8c59-4864-b724-e5f88ca6908e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/68384321-D14F-4525-B301-4BF38B75B9E1%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[gjr80]

What does the error trace show? WeeWX shouldn't be concerned about extra config 
options. We don't need to re-invent wheels, forecast extension (among others) 
already uses a WU API key, not appropriate to use that config option here but 
using a consistent config option naming convention makes it easier for users.

Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/b069c856-8c59-4864-b724-e5f88ca6908e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Well shoot.

Found an issue that surfaced only upon weewx restart.
Don't add the apiKey to weewx.conf.
Found out engine.py is picky about properties it isn't expecting to find in the 
weewx.conf file.  Thought it would just ignore them.
So I guess I need to extend the dictionary of expected properties now, too.  :S

But you can still run this new wunderfixer on the side by passing the --apikey 
argument.

You could for example just call it wunderfixer_apikey and always pass your key 
on command-line:

$ wunderfixer_apikey --apikey=1234567890123456789012345678901
(With your real key there)...

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 24, 2019, at 5:01 PM, Leon Shaner  wrote:
> 
> Hi, WeeWX'ers!  =D
> 
> For those on WeeWX 4.0 / development, ONLY...
> I am done with changing wunderfixer to use the new wunderground API KEY way 
> of doing the WU query to see what records they have for comparison to those 
> in the archive DB.   Yay!  =D
> 
> Please see the comments in the pull request over here:
> 
> https://github.com/weewx/weewx/pull/416
> 
> And until that gets merged you can grab a copy from here -- again this is the 
> 4.0 / development version, ONLY:
> 
> https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer
> 
> 
> Now, I'm off to work on the backport to WeeWX 3.9.1.
> Fun fun!  =D
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 24, 2019, at 1:35 PM, Leon Shaner  wrote:
>> 
>> FYI, all, I am currently hot on the trail to converting the wunderfixer 
>> query to use the new API instead.  The new API is going to be much easier 
>> than the old query, because the output is neatly formatted as JSON, and 
>> crucially for each record they even include the "epoch" i.e. Unix timestamp, 
>> which is all we care about in that query anyway.
>> The use of "epoch" means I won't even have to convert the date strings to do 
>> the matching.  =D
>> Even the input date parameter is cleaner a la 20190524.  =D
>> 
>> What I am slightly undecided about is how/where to store the API key.
>> I think for compatibility reasons I'll add a new field in the weewx.conf for 
>> the API key.
>> That way existing code based on a password can still use that method, and 
>> over time newer code can use the API key explicitly instead of a password.
>> 
>> Once I have the new wunderfixer working, for weewx 4.0, I'll work to 
>> backport the changes to weewx 3.9.1.
>> 
>> Hopefully I don't forget also to update wee_debug to obfuscate the API key, 
>> before I'm done with all of this.  LOL
>> 
>> Docs on the new API are over here...  This is for people like us who have a 
>> PWS, as opposed to other people who have to pay for API usage:
>> 
>> https://docs.google.com/document/d/1eKCnKXI9xnoMGRRzOL1xPCBihNV2rOet08qpE_gArAY/edit
>> The API Key is generated in the "Member Settings" section of 
>> wunderground.com ("My Profile" > "Member Settings" > "API Keys").
>> 
>> 
>> With any luck I'll have this banged out in a few hours.  =D
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 23, 2019, at 9:33 PM, Leon Shaner  wrote:
>>> 
>>> I am in contact with IBM.  This whole interface is entirely for THEIR 
>>> benefit.
>>> I am persistent SOB. It'll be fixed.  LOL
>>> 
>>> Regards,
>>> \Leon
>>> --
>>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>> 
 On May 23, 2019, at 9:22 PM, Jarom Hatch  wrote:
 
 Even after that none of the backfilled data made it in.  If they are no 
 longer accepting backfilled data then fixing wunderfixer might not be 
 possible.
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 "weewx-user" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to weewx-user+unsubscr...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/weewx-user/76f9aa83-37ac-4014-aa04-7f675d63a6e1%40googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to weewx-user+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/weewx-user/36EA4E1E-EEAE-4210-A18E-5024653E6F09%40isylum.org.
>>> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/D2429CA0-DD88-4316-8B8C-2D863F798477%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Hi, WeeWX'ers!  =D

For those on WeeWX 4.0 / development, ONLY...
I am done with changing wunderfixer to use the new wunderground API KEY way of 
doing the WU query to see what records they have for comparison to those in the 
archive DB.   Yay!  =D

Please see the comments in the pull request over here:

https://github.com/weewx/weewx/pull/416

And until that gets merged you can grab a copy from here -- again this is the 
4.0 / development version, ONLY:

https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer


Now, I'm off to work on the backport to WeeWX 3.9.1.
Fun fun!  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 24, 2019, at 1:35 PM, Leon Shaner  wrote:
> 
> FYI, all, I am currently hot on the trail to converting the wunderfixer query 
> to use the new API instead.  The new API is going to be much easier than the 
> old query, because the output is neatly formatted as JSON, and crucially for 
> each record they even include the "epoch" i.e. Unix timestamp, which is all 
> we care about in that query anyway.
> The use of "epoch" means I won't even have to convert the date strings to do 
> the matching.  =D
> Even the input date parameter is cleaner a la 20190524.  =D
> 
> What I am slightly undecided about is how/where to store the API key.
> I think for compatibility reasons I'll add a new field in the weewx.conf for 
> the API key.
> That way existing code based on a password can still use that method, and 
> over time newer code can use the API key explicitly instead of a password.
> 
> Once I have the new wunderfixer working, for weewx 4.0, I'll work to backport 
> the changes to weewx 3.9.1.
> 
> Hopefully I don't forget also to update wee_debug to obfuscate the API key, 
> before I'm done with all of this.  LOL
> 
> Docs on the new API are over here...  This is for people like us who have a 
> PWS, as opposed to other people who have to pay for API usage:
> 
> https://docs.google.com/document/d/1eKCnKXI9xnoMGRRzOL1xPCBihNV2rOet08qpE_gArAY/edit
> The API Key is generated in the "Member Settings" section of wunderground.com 
> ("My Profile" > "Member Settings" > "API Keys").
> 
> 
> With any luck I'll have this banged out in a few hours.  =D
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 23, 2019, at 9:33 PM, Leon Shaner  wrote:
>> 
>> I am in contact with IBM.  This whole interface is entirely for THEIR 
>> benefit.
>> I am persistent SOB. It'll be fixed.  LOL
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 23, 2019, at 9:22 PM, Jarom Hatch  wrote:
>>> 
>>> Even after that none of the backfilled data made it in.  If they are no 
>>> longer accepting backfilled data then fixing wunderfixer might not be 
>>> possible.
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to weewx-user+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/weewx-user/76f9aa83-37ac-4014-aa04-7f675d63a6e1%40googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/36EA4E1E-EEAE-4210-A18E-5024653E6F09%40isylum.org.
>> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/2C6BD081-290F-4FE7-81F7-76D9FCE230DF%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

FYI, all, I am currently hot on the trail to converting the wunderfixer query 
to use the new API instead.  The new API is going to be much easier than the 
old query, because the output is neatly formatted as JSON, and crucially for 
each record they even include the "epoch" i.e. Unix timestamp, which is all we 
care about in that query anyway.
The use of "epoch" means I won't even have to convert the date strings to do 
the matching.  =D
Even the input date parameter is cleaner a la 20190524.  =D

What I am slightly undecided about is how/where to store the API key.
I think for compatibility reasons I'll add a new field in the weewx.conf for 
the API key.
That way existing code based on a password can still use that method, and over 
time newer code can use the API key explicitly instead of a password.

Once I have the new wunderfixer working, for weewx 4.0, I'll work to backport 
the changes to weewx 3.9.1.

Hopefully I don't forget also to update wee_debug to obfuscate the API key, 
before I'm done with all of this.  LOL

Docs on the new API are over here...  This is for people like us who have a 
PWS, as opposed to other people who have to pay for API usage:

https://docs.google.com/document/d/1eKCnKXI9xnoMGRRzOL1xPCBihNV2rOet08qpE_gArAY/edit
The API Key is generated in the "Member Settings" section of wunderground.com 
("My Profile" > "Member Settings" > "API Keys").


With any luck I'll have this banged out in a few hours.  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 23, 2019, at 9:33 PM, Leon Shaner  wrote:
> 
> I am in contact with IBM.  This whole interface is entirely for THEIR benefit.
> I am persistent SOB. It'll be fixed.  LOL
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 23, 2019, at 9:22 PM, Jarom Hatch  wrote:
>> 
>> Even after that none of the backfilled data made it in.  If they are no 
>> longer accepting backfilled data then fixing wunderfixer might not be 
>> possible.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/76f9aa83-37ac-4014-aa04-7f675d63a6e1%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/36EA4E1E-EEAE-4210-A18E-5024653E6F09%40isylum.org.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/6095CF30-8D08-4A43-A445-0265FEA0850D%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

I am in contact with IBM.  This whole intersection is entirely for THEIR 
benefit.
I am persistent SOB. It'll be fixed.  LOL

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 23, 2019, at 9:22 PM, Jarom Hatch  wrote:
> 
> Even after that none of the backfilled data made it in.  If they are no 
> longer accepting backfilled data then fixing wunderfixer might not be 
> possible.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/76f9aa83-37ac-4014-aa04-7f675d63a6e1%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/36EA4E1E-EEAE-4210-A18E-5024653E6F09%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Jarom Hatch]

Even after that none of the backfilled data made it in.  If they are no longer 
accepting backfilled data then fixing wunderfixer might not be possible.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/76f9aa83-37ac-4014-aa04-7f675d63a6e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Jarom Hatch]

Yesterday when I was able to download the csv I saved a copy of it.  I put 
it up on a site hosted by me and changed the URL in wunderfixer to point 
there instead.  Now it parses it and starts the backfill.  Still isn't 
showing up on their side though.  I'm going to run it about 10 times over 
the next few hours and see what happens.  Could be that Akamai is blocking 
the backfill.  Or they have totally jacked up their backend.
On Thursday, May 23, 2019 at 10:10:28 AM UTC-6, Leon Shaner wrote:

Yeah, I have found that it is normal for WU to lazily process re-uploaded 
> records, and it seems as if they drop them...as if some backend process is 
> dying.
>
> I found through trial and error that after a weewx "outage" I need to 
> re-upload exactly 10 times, but not back to back.  I do one upload every 20 
> minutes, which means spreading the 10 re-uploads out over a 200 minute 
> period.
> After that wunderfixer reports no missing records.
> Well, at least it used to work before all these 404's and 503's.  :-(
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/dbe70c45-d4dd-4718-b5af-c9b5c1a7de34%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Jarom Hatch]

I haven't seen a 403 since you changed the user agent.

On Thursday, May 23, 2019 at 10:10:28 AM UTC-6, Leon Shaner wrote:
>
> Jarom,
>
> So you got 503's with my updated code, but not 403's?
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/c9122812-155c-465a-a003-e45d9dff623b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Jarom Hatch]

I haven't seen a 403 since you changed the user agent.

On Thursday, May 23, 2019 at 10:10:28 AM UTC-6, Leon Shaner wrote:
>
> Jarom,
>
> So you got 503's with my updated code, but not 403's?
>
> Yeah, I have found that it is normal for WU to lazily process re-uploaded 
> records, and it seems as if they drop them...as if some backend process is 
> dying.
>
> I found through trial and error that after a weewx "outage" I need to 
> re-upload exactly 10 times, but not back to back.  I do one upload every 20 
> minutes, which means spreading the 10 re-uploads out over a 200 minute 
> period.
> After that wunderfixer reports no missing records.
> Well, at least it used to work before all these 404's and 503's.  :-(
>
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>
> On May 23, 2019, at 11:06 AM, Jarom Hatch > 
> wrote:
>
> Those 503's are coming from Akamai.  When I have used Akamai in the past 
> those errors are because the origin is not responding correctly.
>
> Related sorta, when I took out the code to pull the timestamps and just 
> force-ran all my records as a big backfill I got success messages back 
> however none of the records actually made their way in.
>
> On Thursday, May 23, 2019 at 8:27:37 AM UTC-6, Leon Shaner wrote:
>>
>> Jarom,
>>
>> Thanks so much!  I see what I did wrong and I was able to make a stubbed 
>> down version for basic testing to prove it's at least trying to connect.
>>
>> Same location (for WeeWX 3.9.1):
>>
>>
>> https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
>>
>> The thing is, I'm still constantly getting 404 (Not Found) even with 
>> CURL, and just a bit ago the site started throwing 503 (Service Not 
>> Available).
>> So...  It's kinda hard to test under these conditions.   But as long as 
>> you don't get a 403, then at least my User-Agent "hack" will be "proven."   
>> :-/
>>
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>
>> On May 23, 2019, at 1:57 AM, Jarom Hatch  wrote:
>>
>> I tried the 3.9.1 version and I get Could not get Weather Underground 
>> data. Exiting.
>>
>> Curl still works, even for yesterday's data.  Tracing the script it 
>> doesn't appear to be actually attempting the download.  
>>
>>
>> On Wednesday, May 22, 2019 at 7:03:49 PM UTC-6, Leon Shaner wrote:
>>>
>>> Say, we need a tester who is still on 3.9.1 or there abouts to try this 
>>> out:
>>>
>>>
>>> https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
>>>
>>> Can't do anything to workaround WU's sporadic 404 and 503 errors, but at 
>>> least the 403 error should be gone.
>>>
>>> I was able to test the 4.0 / development version myself on both Python2 
>>> and 3, so hopefully Tom will merge that soon.  It's over here if you are 
>>> impatient.  =D
>>>
>>>
>>> https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer
>>>
>>> Regards,
>>> \Leon
>>> --
>>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>>
>>> On May 22, 2019, at 7:24 PM, Leon Shaner  wrote:
>>>
>>> Hey WeeWX'ers!!!  =D
>>>
>>> I have a fix in the hopper.
>>>
>>> There's nothing that can be done for the occasional HTTP 404, or even 
>>> 503's I am now seeing, but the HTTP 403 was due to a change on WU's part 
>>> where they are rejecting certain HTTP User-Agent strings.  The fact that 
>>> they are putting Akamai in the middle is almost certainly a great thing re: 
>>> their scalability issues; however, they probably inherited some default 
>>> settings that filter "bots" and malware and such, which is likely why the 
>>> HTTP User-Agent now matters.
>>>
>>> I have set the User-Agent to "CURL" and it works.
>>> I have set it to "Mozilla" and it works.  I'm going with that one, since 
>>> it means Mosaic Killer, both of which were among the the very first 
>>> User-Agents I ever worked with, circa 1993 back before there was such as 
>>> thing as Netscape.  =D
>>>
>>> /ye-olde-farte mode off  ;-)
>>>
>>> My testing has so far been under Python3, but coincidentally (and not a 
>>> causation), WU started throwing HTTP 503's around the time that I tried 
>>> validating my code also under Python2.
>>>
>>> Everything is working against today's date.
>>> It's when I go after yesterday's date that I get the HTTP server error 
>>> 503.
>>>
>>> I expect the 404's and 503's to go away eventually, but at least for now 
>>> I have a fix for the 403 (forbidden)'s, just based on the User-Agent string.
>>>
>>> I'll submit a change for wunderfixer both to the 3.9.x "master" and 
>>> 4.0.x "development" branches in a moment and reply back with direct links 
>>> for anyone who wants a fix sooner.  =D
>>>
>>> Isn't this fun?  =D
>>>
>>> Regards,
>>> \Leon
>>> --
>>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>>
>>> On May 22, 2019, at 4:20 PM, Leon Shaner  wrote:
>>>
>>> I'm still working on this.
>>> CURL is telling me they are not only using https, but also TLSv1.2.
>>> Here is a transcript, in case one of y'all beats me to

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Jarom,

So you got 503's with my updated code, but not 403's?

Yeah, I have found that it is normal for WU to lazily process re-uploaded 
records, and it seems as if they drop them...as if some backend process is 
dying.

I found through trial and error that after a weewx "outage" I need to re-upload 
exactly 10 times, but not back to back.  I do one upload every 20 minutes, 
which means spreading the 10 re-uploads out over a 200 minute period.
After that wunderfixer reports no missing records.
Well, at least it used to work before all these 404's and 503's.  :-(

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 23, 2019, at 11:06 AM, Jarom Hatch  wrote:
> 
> Those 503's are coming from Akamai.  When I have used Akamai in the past 
> those errors are because the origin is not responding correctly.
> 
> Related sorta, when I took out the code to pull the timestamps and just 
> force-ran all my records as a big backfill I got success messages back 
> however none of the records actually made their way in.
> 
>> On Thursday, May 23, 2019 at 8:27:37 AM UTC-6, Leon Shaner wrote:
>> Jarom,
>> 
>> Thanks so much!  I see what I did wrong and I was able to make a stubbed 
>> down version for basic testing to prove it's at least trying to connect.
>> 
>> Same location (for WeeWX 3.9.1):
>> 
>> https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
>> 
>> The thing is, I'm still constantly getting 404 (Not Found) even with CURL, 
>> and just a bit ago the site started throwing 503 (Service Not Available).
>> So...  It's kinda hard to test under these conditions.   But as long as you 
>> don't get a 403, then at least my User-Agent "hack" will be "proven."   :-/
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 23, 2019, at 1:57 AM, Jarom Hatch  wrote:
>>> 
>>> I tried the 3.9.1 version and I get Could not get Weather Underground data. 
>>> Exiting.
>>> 
>>> Curl still works, even for yesterday's data.  Tracing the script it doesn't 
>>> appear to be actually attempting the download.  
>>> 
>>> 
 On Wednesday, May 22, 2019 at 7:03:49 PM UTC-6, Leon Shaner wrote:
 Say, we need a tester who is still on 3.9.1 or there abouts to try this 
 out:
 
 https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
 
 Can't do anything to workaround WU's sporadic 404 and 503 errors, but at 
 least the 403 error should be gone.
 
 I was able to test the 4.0 / development version myself on both Python2 
 and 3, so hopefully Tom will merge that soon.  It's over here if you are 
 impatient.  =D
 
 https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer
 
 Regards,
 \Leon
 --
 Leon Shaner :: Dearborn, Michigan (iPad Pro)
 
> On May 22, 2019, at 7:24 PM, Leon Shaner  wrote:
> 
> Hey WeeWX'ers!!!  =D
> 
> I have a fix in the hopper.
> 
> There's nothing that can be done for the occasional HTTP 404, or even 
> 503's I am now seeing, but the HTTP 403 was due to a change on WU's part 
> where they are rejecting certain HTTP User-Agent strings.  The fact that 
> they are putting Akamai in the middle is almost certainly a great thing 
> re: their scalability issues; however, they probably inherited some 
> default settings that filter "bots" and malware and such, which is likely 
> why the HTTP User-Agent now matters.
> 
> I have set the User-Agent to "CURL" and it works.
> I have set it to "Mozilla" and it works.  I'm going with that one, since 
> it means Mosaic Killer, both of which were among the the very first 
> User-Agents I ever worked with, circa 1993 back before there was such as 
> thing as Netscape.  =D
> 
> /ye-olde-farte mode off  ;-)
> 
> My testing has so far been under Python3, but coincidentally (and not a 
> causation), WU started throwing HTTP 503's around the time that I tried 
> validating my code also under Python2.
> 
> Everything is working against today's date.
> It's when I go after yesterday's date that I get the HTTP server error 
> 503.
> 
> I expect the 404's and 503's to go away eventually, but at least for now 
> I have a fix for the 403 (forbidden)'s, just based on the User-Agent 
> string.
> 
> I'll submit a change for wunderfixer both to the 3.9.x "master" and 4.0.x 
> "development" branches in a moment and reply back with direct links for 
> anyone who wants a fix sooner.  =D
> 
> Isn't this fun?  =D
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 22, 2019, at 4:20 PM, Leon Shaner  wrote:
>> 
>> I'm still working on this.
>> CURL is telling me they are not only using https, but also TLSv1.2.
>> Here is a transcript, in case one of y'all beats me to the fix.  =D

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Jarom Hatch]

Those 503's are coming from Akamai.  When I have used Akamai in the past 
those errors are because the origin is not responding correctly.

Related sorta, when I took out the code to pull the timestamps and just 
force-ran all my records as a big backfill I got success messages back 
however none of the records actually made their way in.

On Thursday, May 23, 2019 at 8:27:37 AM UTC-6, Leon Shaner wrote:
>
> Jarom,
>
> Thanks so much!  I see what I did wrong and I was able to make a stubbed 
> down version for basic testing to prove it's at least trying to connect.
>
> Same location (for WeeWX 3.9.1):
>
> https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
>
> The thing is, I'm still constantly getting 404 (Not Found) even with CURL, 
> and just a bit ago the site started throwing 503 (Service Not Available).
> So...  It's kinda hard to test under these conditions.   But as long as 
> you don't get a 403, then at least my User-Agent "hack" will be "proven."   
> :-/
>
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>
> On May 23, 2019, at 1:57 AM, Jarom Hatch > 
> wrote:
>
> I tried the 3.9.1 version and I get Could not get Weather Underground 
> data. Exiting.
>
> Curl still works, even for yesterday's data.  Tracing the script it 
> doesn't appear to be actually attempting the download.  
>
>
> On Wednesday, May 22, 2019 at 7:03:49 PM UTC-6, Leon Shaner wrote:
>>
>> Say, we need a tester who is still on 3.9.1 or there abouts to try this 
>> out:
>>
>>
>> https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
>>
>> Can't do anything to workaround WU's sporadic 404 and 503 errors, but at 
>> least the 403 error should be gone.
>>
>> I was able to test the 4.0 / development version myself on both Python2 
>> and 3, so hopefully Tom will merge that soon.  It's over here if you are 
>> impatient.  =D
>>
>>
>> https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer
>>
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>
>> On May 22, 2019, at 7:24 PM, Leon Shaner  wrote:
>>
>> Hey WeeWX'ers!!!  =D
>>
>> I have a fix in the hopper.
>>
>> There's nothing that can be done for the occasional HTTP 404, or even 
>> 503's I am now seeing, but the HTTP 403 was due to a change on WU's part 
>> where they are rejecting certain HTTP User-Agent strings.  The fact that 
>> they are putting Akamai in the middle is almost certainly a great thing re: 
>> their scalability issues; however, they probably inherited some default 
>> settings that filter "bots" and malware and such, which is likely why the 
>> HTTP User-Agent now matters.
>>
>> I have set the User-Agent to "CURL" and it works.
>> I have set it to "Mozilla" and it works.  I'm going with that one, since 
>> it means Mosaic Killer, both of which were among the the very first 
>> User-Agents I ever worked with, circa 1993 back before there was such as 
>> thing as Netscape.  =D
>>
>> /ye-olde-farte mode off  ;-)
>>
>> My testing has so far been under Python3, but coincidentally (and not a 
>> causation), WU started throwing HTTP 503's around the time that I tried 
>> validating my code also under Python2.
>>
>> Everything is working against today's date.
>> It's when I go after yesterday's date that I get the HTTP server error 
>> 503.
>>
>> I expect the 404's and 503's to go away eventually, but at least for now 
>> I have a fix for the 403 (forbidden)'s, just based on the User-Agent string.
>>
>> I'll submit a change for wunderfixer both to the 3.9.x "master" and 4.0.x 
>> "development" branches in a moment and reply back with direct links for 
>> anyone who wants a fix sooner.  =D
>>
>> Isn't this fun?  =D
>>
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>
>> On May 22, 2019, at 4:20 PM, Leon Shaner  wrote:
>>
>> I'm still working on this.
>> CURL is telling me they are not only using https, but also TLSv1.2.
>> Here is a transcript, in case one of y'all beats me to the fix.  =D
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/DA01E425-B99A-4959-8FB2-B564A61B3E77%40isylum.org
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>> 
>>
>>
>>
>>
>> Working from here:
>> https://docs.python.org/2/library/ssl.html
>>
>> So far I have tried this, to no avail.
>> Really just doing the "import ssl" and using https in the URL, and adding 
>> context=ssl_context to the urllib.request.
>>
>> A snippet of that looks as follows, but still getting 403 forbidden.  :-(
>>
>> # For new WU interface which uses SSL 

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Jarom Hatch]

Sad, now I'm getting the 503.  I'll keep trying.

On Thursday, May 23, 2019 at 8:27:37 AM UTC-6, Leon Shaner wrote:
>
> Jarom,
>
> Thanks so much!  I see what I did wrong and I was able to make a stubbed 
> down version for basic testing to prove it's at least trying to connect.
>
> Same location (for WeeWX 3.9.1):
>
> https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
>
> The thing is, I'm still constantly getting 404 (Not Found) even with CURL, 
> and just a bit ago the site started throwing 503 (Service Not Available).
> So...  It's kinda hard to test under these conditions.   But as long as 
> you don't get a 403, then at least my User-Agent "hack" will be "proven."   
> :-/
>
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>
> On May 23, 2019, at 1:57 AM, Jarom Hatch > 
> wrote:
>
> I tried the 3.9.1 version and I get Could not get Weather Underground 
> data. Exiting.
>
> Curl still works, even for yesterday's data.  Tracing the script it 
> doesn't appear to be actually attempting the download.  
>
>
> On Wednesday, May 22, 2019 at 7:03:49 PM UTC-6, Leon Shaner wrote:
>>
>> Say, we need a tester who is still on 3.9.1 or there abouts to try this 
>> out:
>>
>>
>> https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
>>
>> Can't do anything to workaround WU's sporadic 404 and 503 errors, but at 
>> least the 403 error should be gone.
>>
>> I was able to test the 4.0 / development version myself on both Python2 
>> and 3, so hopefully Tom will merge that soon.  It's over here if you are 
>> impatient.  =D
>>
>>
>> https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer
>>
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>
>> On May 22, 2019, at 7:24 PM, Leon Shaner  wrote:
>>
>> Hey WeeWX'ers!!!  =D
>>
>> I have a fix in the hopper.
>>
>> There's nothing that can be done for the occasional HTTP 404, or even 
>> 503's I am now seeing, but the HTTP 403 was due to a change on WU's part 
>> where they are rejecting certain HTTP User-Agent strings.  The fact that 
>> they are putting Akamai in the middle is almost certainly a great thing re: 
>> their scalability issues; however, they probably inherited some default 
>> settings that filter "bots" and malware and such, which is likely why the 
>> HTTP User-Agent now matters.
>>
>> I have set the User-Agent to "CURL" and it works.
>> I have set it to "Mozilla" and it works.  I'm going with that one, since 
>> it means Mosaic Killer, both of which were among the the very first 
>> User-Agents I ever worked with, circa 1993 back before there was such as 
>> thing as Netscape.  =D
>>
>> /ye-olde-farte mode off  ;-)
>>
>> My testing has so far been under Python3, but coincidentally (and not a 
>> causation), WU started throwing HTTP 503's around the time that I tried 
>> validating my code also under Python2.
>>
>> Everything is working against today's date.
>> It's when I go after yesterday's date that I get the HTTP server error 
>> 503.
>>
>> I expect the 404's and 503's to go away eventually, but at least for now 
>> I have a fix for the 403 (forbidden)'s, just based on the User-Agent string.
>>
>> I'll submit a change for wunderfixer both to the 3.9.x "master" and 4.0.x 
>> "development" branches in a moment and reply back with direct links for 
>> anyone who wants a fix sooner.  =D
>>
>> Isn't this fun?  =D
>>
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>
>> On May 22, 2019, at 4:20 PM, Leon Shaner  wrote:
>>
>> I'm still working on this.
>> CURL is telling me they are not only using https, but also TLSv1.2.
>> Here is a transcript, in case one of y'all beats me to the fix.  =D
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/DA01E425-B99A-4959-8FB2-B564A61B3E77%40isylum.org
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>> 
>>
>>
>>
>>
>> Working from here:
>> https://docs.python.org/2/library/ssl.html
>>
>> So far I have tried this, to no avail.
>> Really just doing the "import ssl" and using https in the URL, and adding 
>> context=ssl_context to the urllib.request.
>>
>> A snippet of that looks as follows, but still getting 403 forbidden.  :-(
>>
>> # For new WU interface which uses SSL and TLSv1.2
>> import ssl
>>
>> ...
>>
>> _url = "
>> https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=%s"; \
>>"&month=%d&day=%d&year=%d&format=1" % (self.station, 
>> dayRequested_tt[1],
>>   dayRe

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Jarom,

Thanks so much!  I see what I did wrong and I was able to make a stubbed down 
version for basic testing to prove it's at least trying to connect.

Same location (for WeeWX 3.9.1):

https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer

The thing is, I'm still constantly getting 404 (Not Found) even with CURL, and 
just a bit ago the site started throwing 503 (Service Not Available).
So...  It's kinda hard to test under these conditions.   But as long as you 
don't get a 403, then at least my User-Agent "hack" will be "proven."   :-/

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 23, 2019, at 1:57 AM, Jarom Hatch  wrote:
> 
> I tried the 3.9.1 version and I get Could not get Weather Underground data. 
> Exiting.
> 
> Curl still works, even for yesterday's data.  Tracing the script it doesn't 
> appear to be actually attempting the download.  
> 
> 
>> On Wednesday, May 22, 2019 at 7:03:49 PM UTC-6, Leon Shaner wrote:
>> Say, we need a tester who is still on 3.9.1 or there abouts to try this out:
>> 
>> https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
>> 
>> Can't do anything to workaround WU's sporadic 404 and 503 errors, but at 
>> least the 403 error should be gone.
>> 
>> I was able to test the 4.0 / development version myself on both Python2 and 
>> 3, so hopefully Tom will merge that soon.  It's over here if you are 
>> impatient.  =D
>> 
>> https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 22, 2019, at 7:24 PM, Leon Shaner  wrote:
>>> 
>>> Hey WeeWX'ers!!!  =D
>>> 
>>> I have a fix in the hopper.
>>> 
>>> There's nothing that can be done for the occasional HTTP 404, or even 503's 
>>> I am now seeing, but the HTTP 403 was due to a change on WU's part where 
>>> they are rejecting certain HTTP User-Agent strings.  The fact that they are 
>>> putting Akamai in the middle is almost certainly a great thing re: their 
>>> scalability issues; however, they probably inherited some default settings 
>>> that filter "bots" and malware and such, which is likely why the HTTP 
>>> User-Agent now matters.
>>> 
>>> I have set the User-Agent to "CURL" and it works.
>>> I have set it to "Mozilla" and it works.  I'm going with that one, since it 
>>> means Mosaic Killer, both of which were among the the very first 
>>> User-Agents I ever worked with, circa 1993 back before there was such as 
>>> thing as Netscape.  =D
>>> 
>>> /ye-olde-farte mode off  ;-)
>>> 
>>> My testing has so far been under Python3, but coincidentally (and not a 
>>> causation), WU started throwing HTTP 503's around the time that I tried 
>>> validating my code also under Python2.
>>> 
>>> Everything is working against today's date.
>>> It's when I go after yesterday's date that I get the HTTP server error 503.
>>> 
>>> I expect the 404's and 503's to go away eventually, but at least for now I 
>>> have a fix for the 403 (forbidden)'s, just based on the User-Agent string.
>>> 
>>> I'll submit a change for wunderfixer both to the 3.9.x "master" and 4.0.x 
>>> "development" branches in a moment and reply back with direct links for 
>>> anyone who wants a fix sooner.  =D
>>> 
>>> Isn't this fun?  =D
>>> 
>>> Regards,
>>> \Leon
>>> --
>>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>> 
 On May 22, 2019, at 4:20 PM, Leon Shaner  wrote:
 
 I'm still working on this.
 CURL is telling me they are not only using https, but also TLSv1.2.
 Here is a transcript, in case one of y'all beats me to the fix.  =D
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 "weewx-user" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to weewx...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/weewx-user/DA01E425-B99A-4959-8FB2-B564A61B3E77%40isylum.org.
 For more options, visit https://groups.google.com/d/optout.
 
 
 
 
 Working from here:
 https://docs.python.org/2/library/ssl.html
 
 So far I have tried this, to no avail.
 Really just doing the "import ssl" and using https in the URL, and adding 
 context=ssl_context to the urllib.request.
 
 A snippet of that looks as follows, but still getting 403 forbidden.  :-(
 
 # For new WU interface which uses SSL and TLSv1.2
 import ssl
 
 ...
 
 _url = 
 "https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=%s"; \
"&month=%d&day=%d&year=%d&format=1" % (self.station, 
 dayRequested_tt[1],
   dayRequested_tt[2], 
 dayRequested_tt[0])
 
 # specify TLSv1.2 and SSLv2, but not SSLv3
 ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
   

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Jarom Hatch]

I tried the 3.9.1 version and I get Could not get Weather Underground data. 
Exiting.

Curl still works, even for yesterday's data.  Tracing the script it doesn't 
appear to be actually attempting the download.  


On Wednesday, May 22, 2019 at 7:03:49 PM UTC-6, Leon Shaner wrote:
>
> Say, we need a tester who is still on 3.9.1 or there abouts to try this 
> out:
>
> https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
>
> Can't do anything to workaround WU's sporadic 404 and 503 errors, but at 
> least the 403 error should be gone.
>
> I was able to test the 4.0 / development version myself on both Python2 
> and 3, so hopefully Tom will merge that soon.  It's over here if you are 
> impatient.  =D
>
>
> https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer
>
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>
> On May 22, 2019, at 7:24 PM, Leon Shaner > 
> wrote:
>
> Hey WeeWX'ers!!!  =D
>
> I have a fix in the hopper.
>
> There's nothing that can be done for the occasional HTTP 404, or even 
> 503's I am now seeing, but the HTTP 403 was due to a change on WU's part 
> where they are rejecting certain HTTP User-Agent strings.  The fact that 
> they are putting Akamai in the middle is almost certainly a great thing re: 
> their scalability issues; however, they probably inherited some default 
> settings that filter "bots" and malware and such, which is likely why the 
> HTTP User-Agent now matters.
>
> I have set the User-Agent to "CURL" and it works.
> I have set it to "Mozilla" and it works.  I'm going with that one, since 
> it means Mosaic Killer, both of which were among the the very first 
> User-Agents I ever worked with, circa 1993 back before there was such as 
> thing as Netscape.  =D
>
> /ye-olde-farte mode off  ;-)
>
> My testing has so far been under Python3, but coincidentally (and not a 
> causation), WU started throwing HTTP 503's around the time that I tried 
> validating my code also under Python2.
>
> Everything is working against today's date.
> It's when I go after yesterday's date that I get the HTTP server error 503.
>
> I expect the 404's and 503's to go away eventually, but at least for now I 
> have a fix for the 403 (forbidden)'s, just based on the User-Agent string.
>
> I'll submit a change for wunderfixer both to the 3.9.x "master" and 4.0.x 
> "development" branches in a moment and reply back with direct links for 
> anyone who wants a fix sooner.  =D
>
> Isn't this fun?  =D
>
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>
> On May 22, 2019, at 4:20 PM, Leon Shaner > 
> wrote:
>
> I'm still working on this.
> CURL is telling me they are not only using https, but also TLSv1.2.
> Here is a transcript, in case one of y'all beats me to the fix.  =D
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/DA01E425-B99A-4959-8FB2-B564A61B3E77%40isylum.org
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
> 
>
>
>
>
> Working from here:
> https://docs.python.org/2/library/ssl.html
>
> So far I have tried this, to no avail.
> Really just doing the "import ssl" and using https in the URL, and adding 
> context=ssl_context to the urllib.request.
>
> A snippet of that looks as follows, but still getting 403 forbidden.  :-(
>
> # For new WU interface which uses SSL and TLSv1.2
> import ssl
>
> ...
>
> _url = "
> https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=%s"; \
>"&month=%d&day=%d&year=%d&format=1" % (self.station, 
> dayRequested_tt[1],
>   dayRequested_tt[2], 
> dayRequested_tt[0])
>
> # specify TLSv1.2 and SSLv2, but not SSLv3
> ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
> ssl_context.options |= ssl.PROTOCOL_SSLv23
> ssl_context.options |= ssl.OP_NO_SSLv3
>
> try :
> # Hit the weather underground site:
> _wudata = urllib.request.urlopen(_url, context=ssl_context)
>
>
>
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>
> On May 22, 2019, at 2:42 PM, Leon Shaner > 
> wrote:
>
> Jarom,
>
> CURL is pretty sophisticated in its ability to emulate browser state in 
> pretty much any way but JavaScript.  When it worked this morning, I saw 
> some cookies were involved.
> It may well be that the python way isn't handling that part.
> I don't know enough about how python fetches pages to work that out, but I 
> am very familiar with CURL, so if I can find a path that works 
> consistently, then I'll go b

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Say, we need a tester who is still on 3.9.1 or there abouts to try this out:

https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer

Can't do anything to workaround WU's sporadic 404 and 503 errors, but at least 
the 403 error should be gone.

I was able to test the 4.0 / development version myself on both Python2 and 3, 
so hopefully Tom will merge that soon.  It's over here if you are impatient.  =D

https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 22, 2019, at 7:24 PM, Leon Shaner  wrote:
> 
> Hey WeeWX'ers!!!  =D
> 
> I have a fix in the hopper.
> 
> There's nothing that can be done for the occasional HTTP 404, or even 503's I 
> am now seeing, but the HTTP 403 was due to a change on WU's part where they 
> are rejecting certain HTTP User-Agent strings.  The fact that they are 
> putting Akamai in the middle is almost certainly a great thing re: their 
> scalability issues; however, they probably inherited some default settings 
> that filter "bots" and malware and such, which is likely why the HTTP 
> User-Agent now matters.
> 
> I have set the User-Agent to "CURL" and it works.
> I have set it to "Mozilla" and it works.  I'm going with that one, since it 
> means Mosaic Killer, both of which were among the the very first User-Agents 
> I ever worked with, circa 1993 back before there was such as thing as 
> Netscape.  =D
> 
> /ye-olde-farte mode off  ;-)
> 
> My testing has so far been under Python3, but coincidentally (and not a 
> causation), WU started throwing HTTP 503's around the time that I tried 
> validating my code also under Python2.
> 
> Everything is working against today's date.
> It's when I go after yesterday's date that I get the HTTP server error 503.
> 
> I expect the 404's and 503's to go away eventually, but at least for now I 
> have a fix for the 403 (forbidden)'s, just based on the User-Agent string.
> 
> I'll submit a change for wunderfixer both to the 3.9.x "master" and 4.0.x 
> "development" branches in a moment and reply back with direct links for 
> anyone who wants a fix sooner.  =D
> 
> Isn't this fun?  =D
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 22, 2019, at 4:20 PM, Leon Shaner  wrote:
>> 
>> I'm still working on this.
>> CURL is telling me they are not only using https, but also TLSv1.2.
>> Here is a transcript, in case one of y'all beats me to the fix.  =D
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/DA01E425-B99A-4959-8FB2-B564A61B3E77%40isylum.org.
>> For more options, visit https://groups.google.com/d/optout.
>> 
>> 
>> 
>> 
>> Working from here:
>> https://docs.python.org/2/library/ssl.html
>> 
>> So far I have tried this, to no avail.
>> Really just doing the "import ssl" and using https in the URL, and adding 
>> context=ssl_context to the urllib.request.
>> 
>> A snippet of that looks as follows, but still getting 403 forbidden.  :-(
>> 
>> # For new WU interface which uses SSL and TLSv1.2
>> import ssl
>> 
>> ...
>> 
>> _url = 
>> "https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=%s"; \
>>"&month=%d&day=%d&year=%d&format=1" % (self.station, 
>> dayRequested_tt[1],
>>   dayRequested_tt[2], 
>> dayRequested_tt[0])
>> 
>> # specify TLSv1.2 and SSLv2, but not SSLv3
>> ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
>> ssl_context.options |= ssl.PROTOCOL_SSLv23
>> ssl_context.options |= ssl.OP_NO_SSLv3
>> 
>> try :
>> # Hit the weather underground site:
>> _wudata = urllib.request.urlopen(_url, context=ssl_context)
>> 
>> 
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 22, 2019, at 2:42 PM, Leon Shaner  wrote:
>>> 
>>> Jarom,
>>> 
>>> CURL is pretty sophisticated in its ability to emulate browser state in 
>>> pretty much any way but JavaScript.  When it worked this morning, I saw 
>>> some cookies were involved.
>>> It may well be that the python way isn't handling that part.
>>> I don't know enough about how python fetches pages to work that out, but I 
>>> am very familiar with CURL, so if I can find a path that works 
>>> consistently, then I'll go back to the python to see about how to implement 
>>> same.
>>> 
>>> I was getting 404's in the browser even, when I looked at it earlier.
>>> 
>>> I'll keep working on it, but not too hard, so as to not get on their radar 
>>> in any unwanted sort of way.  ;-)
>>> 
>>> Regards,
>>> \Leon
>>> --
>>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>> 
 On

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Hey WeeWX'ers!!!  =D

I have a fix in the hopper.

There's nothing that can be done for the occasional HTTP 404, or even 503's I 
am now seeing, but the HTTP 403 was due to a change on WU's part where they are 
rejecting certain HTTP User-Agent strings.  The fact that they are putting 
Akamai in the middle is almost certainly a great thing re: their scalability 
issues; however, they probably inherited some default settings that filter 
"bots" and malware and such, which is likely why the HTTP User-Agent now 
matters.

I have set the User-Agent to "CURL" and it works.
I have set it to "Mozilla" and it works.  I'm going with that one, since it 
means Mosaic Killer, both of which were among the the very first User-Agents I 
ever worked with, circa 1993 back before there was such as thing as Netscape.  
=D

/ye-olde-farte mode off  ;-)

My testing has so far been under Python3, but coincidentally (and not a 
causation), WU started throwing HTTP 503's around the time that I tried 
validating my code also under Python2.

Everything is working against today's date.
It's when I go after yesterday's date that I get the HTTP server error 503.

I expect the 404's and 503's to go away eventually, but at least for now I have 
a fix for the 403 (forbidden)'s, just based on the User-Agent string.

I'll submit a change for wunderfixer both to the 3.9.x "master" and 4.0.x 
"development" branches in a moment and reply back with direct links for anyone 
who wants a fix sooner.  =D

Isn't this fun?  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 22, 2019, at 4:20 PM, Leon Shaner  wrote:
> 
> I'm still working on this.
> CURL is telling me they are not only using https, but also TLSv1.2.
> Here is a transcript, in case one of y'all beats me to the fix.  =D
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/DA01E425-B99A-4959-8FB2-B564A61B3E77%40isylum.org.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> 
> 
> Working from here:
> https://docs.python.org/2/library/ssl.html
> 
> So far I have tried this, to no avail.
> Really just doing the "import ssl" and using https in the URL, and adding 
> context=ssl_context to the urllib.request.
> 
> A snippet of that looks as follows, but still getting 403 forbidden.  :-(
> 
> # For new WU interface which uses SSL and TLSv1.2
> import ssl
> 
> ...
> 
> _url = 
> "https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=%s"; \
>"&month=%d&day=%d&year=%d&format=1" % (self.station, 
> dayRequested_tt[1],
>   dayRequested_tt[2], 
> dayRequested_tt[0])
> 
> # specify TLSv1.2 and SSLv2, but not SSLv3
> ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
> ssl_context.options |= ssl.PROTOCOL_SSLv23
> ssl_context.options |= ssl.OP_NO_SSLv3
> 
> try :
> # Hit the weather underground site:
> _wudata = urllib.request.urlopen(_url, context=ssl_context)
> 
> 
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 22, 2019, at 2:42 PM, Leon Shaner  wrote:
>> 
>> Jarom,
>> 
>> CURL is pretty sophisticated in its ability to emulate browser state in 
>> pretty much any way but JavaScript.  When it worked this morning, I saw some 
>> cookies were involved.
>> It may well be that the python way isn't handling that part.
>> I don't know enough about how python fetches pages to work that out, but I 
>> am very familiar with CURL, so if I can find a path that works consistently, 
>> then I'll go back to the python to see about how to implement same.
>> 
>> I was getting 404's in the browser even, when I looked at it earlier.
>> 
>> I'll keep working on it, but not too hard, so as to not get on their radar 
>> in any unwanted sort of way.  ;-)
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 22, 2019, at 2:04 PM, Jarom Hatch  wrote:
>>> 
>>> Interesting, using curl sometimes I can it fine, but wunderfixer is always 
>>> getting a 403 Forbidden, as if it is actively being blocked...  When it 
>>> doesn't work in curl I get `HTTP/1.1 404 Not Found` and when it does work I 
>>> get `HTTP/1.1 200 OK`.  Curl never gets a 403 error.
>>> 
 On Wednesday, May 22, 2019 at 11:48:08 AM UTC-6, Jarom Hatch wrote:
 I was able to get it to work twice in my web browser, but as you said, it 
 is sporadic.  I don't ever recall them using Akamai before so that may 
 very well be a contributing factor.
 
 I wonder if we can find out the origin address and see what happens if we 
 can bypass Akamai...
 
> On Wednesday, May 22, 2019 at 7:35:18 AM UTC-6

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Doug Bo]

Hi Leon,

Agreed, I realize my post is a bit off topic but if WU continues to be less 
than reliable then perhaps other solutions might be viable?
Thanks for the reply, perhaps a new thread needs to be started by someone 
smarter than me about historical data search.

Good luck with your research & fix.  I'm a big WU fan and am sad they are 
slipping.  Thank you for trying to resolve the issue.  :)  

Thanks again,
Doug B.

On Wednesday, May 22, 2019 at 1:41:08 PM UTC-7, Leon Shaner wrote:
>
> Hey, Doug,
> Creating a front end to search your own local data would certainly be 
> doable.
> For sure, directing end-users to our own sites and letting them search our 
> historical data would be useful.
>
> But that wouldn't fix the problem at hand.  :-/
> The wunderfixer utility is there to query what data WU has on record from 
> your station and to re-upload any missing data.
> Basically if we can't fix this then it means WU will often be missing data 
> from time to time for various reasons.
> Really only WU should care and end-users who go to their site to look at 
> historical data.   So yeah, they should be invented to help, since they're 
> the ones missing the data, and the less reliable their data, the more 
> likely even end-users will begin to abandon using them.
>
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>
> On May 22, 2019, at 4:33 PM, Doug Bo > 
> wrote:
>
> Sort of a noob question or maybe a feature request: how about adding a web 
> interface to search the database for historic weather data?  I grab the 
> wunderground data that I have uploaded to add to various farm records a few 
> months or even a year after the fact.  If WU is getting worse perhaps I 
> need another solution?
>
> Thanks,
> Doug B.
>
> On Tuesday, May 21, 2019 at 11:09:14 AM UTC-7, tds wrote:
>>
>> For the last three days, my daily cron job of wunderfixer have returned 
>> "503 Unavaialbe" (May 18), "404 Not Found" (May 19) and "403 Forbidden" 
>> (May 20).
>>
>> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/22591326-058f-4717-bdb4-2f91cd099e68%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/cab1516c-06fb-4dcf-80df-184565dc1db5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Hey, Doug,
Creating a front end to search your own local data would certainly be doable.
For sure, directing end-users to our own sites and letting them search our 
historical data would be useful.

But that wouldn't fix the problem at hand.  :-/
The wunderfixer utility is there to query what data WU has on record from your 
station and to re-upload any missing data.
Basically if we can't fix this then it means WU will often be missing data from 
time to time for various reasons.
Really only WU should care and end-users who go to their site to look at 
historical data.   So yeah, they should be invented to help, since they're the 
ones missing the data, and the less reliable their data, the more likely even 
end-users will begin to abandon using them.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 22, 2019, at 4:33 PM, Doug Bo  wrote:
> 
> Sort of a noob question or maybe a feature request: how about adding a web 
> interface to search the database for historic weather data?  I grab the 
> wunderground data that I have uploaded to add to various farm records a few 
> months or even a year after the fact.  If WU is getting worse perhaps I need 
> another solution?
> 
> Thanks,
> Doug B.
> 
>> On Tuesday, May 21, 2019 at 11:09:14 AM UTC-7, tds wrote:
>> For the last three days, my daily cron job of wunderfixer have returned "503 
>> Unavaialbe" (May 18), "404 Not Found" (May 19) and "403 Forbidden" (May 20).
>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/22591326-058f-4717-bdb4-2f91cd099e68%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/2B320D7A-2F1D-475D-810E-BCD69BB562E8%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

I'm still working on this.CURL is telling me they are not only using https, but also TLSv1.2.Here is a transcript, in case one of y'all beats me to the fix.  =D



-- 
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/DA01E425-B99A-4959-8FB2-B564A61B3E77%40isylum.org.
For more options, visit https://groups.google.com/d/optout.
$ curl -L -v 
"https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1";

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0* 
  Trying 23.7.182.137...
* TCP_NODELAY set
* Connected to www.wunderground.com (23.7.182.137) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [102 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2527 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [148 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0* 
TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=Georgia; L=Atlanta; O=TWC Product and Technology, LLC; 
OU=Digital; CN=www.weather.com
*  start date: Sep 25 00:00:00 2018 GMT
*  expire date: Nov 24 12:00:00 2019 GMT
*  subjectAltName: host "www.wunderground.com" matched cert's 
"*.wunderground.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert ECC Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x6cea8)
} [5 bytes data]
> GET 
> /weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1
>  HTTP/1.1
> Host: www.wunderground.com
> User-Agent: curl/7.52.1
> Accept: */*
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]

  0 00 00 0  0  0 --:--:--  0:00:02 --:--:-- 0
  0 00 00 0  0  0 --:--:--  0:00:03 --:--:-- 0
  0 00 00 0  0  0 --:--:--  0:00:04 --:--:-- 0
  0 00 00 0  0  0 --:--:--  0:00:05 --:--:-- 0< 
HTTP/2 200 
< server: Apache/2.2.15 (CentOS)
< pragma: no-cache
< access-control-allow-origin: http://www.wunderground.com
< access-control-allow-credentials: true
< x-creationtime: 4.786
< content-type: text/html; charset=UTF-8
< cache-control: private, no-cache, must-revalidate
< expires: Wed, 22 May 2019 19:35:18 GMT
< date: Wed, 22 May 2019 19:35:18 GMT
< content-length: 19032
< set-cookie: DT=1558553713:13369:ip-10-226-235-63; path=/; expires=Fri, 
01-Jan-2020 00:00:00 GMT; domain=.wunderground.com
< set-cookie: 
Prefs=FAVS:1|WXSN:1|PWSOBS:1|WPHO:1|PHOT:1|RADC:0|RADALL:0|HIST0:NULL|GIFT:1|PHOTOTHUMBS:50|;
 path=/; expires=Fri, 01-Jan-2020 00:00:00 GMT; domain=.wunderground.com
< set-cookie: speedpin=4G; expires=Wed, 22-May-2019 20:05:18 GMT; path=/; 
domain=.wunderground.com; secure
< set-cookie: 
ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=desktop&X-Origin-Hint=dna&TWC-Network-Type=wifi&TWC-GeoIP-Country=US&TWC-GeoIP-Lat=42.3055&TWC-GeoIP-Long=-83.1724&Akamai-Connection-Speed=1000+&TWC-Privacy=exempt;
 path=/; domain=.wunderground.com; secure
< property-id: drupal-prod
< x-requestsource: AkamaiProdProxy
< twc-privacy: exempt
< twc-geoip-latlong: 42.3055,-83.1724
< twc-geoip-country: US
< twc-device-class: desktop
< twc-locale-group: US
< twc-connection-speed: 4G
< x-requestsource: AkamaiDefaultDNA
< 
{ [105 bytes data]

Time,TemperatureF,DewpointF,PressureIn,WindDirection,WindDirectionDegrees,WindSpeedMPH,WindSpeedGustMPH,Humidity,HourlyPrecipIn,Conditions,Clouds,daily

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Jarom,

CURL is pretty sophisticated in its ability to emulate browser state in pretty 
much any way but JavaScript.  When it worked this morning, I saw some cookies 
were involved.
It may well be that the python way isn't handling that part.
I don't know enough about how python fetches pages to work that out, but I am 
very familiar with CURL, so if I can find a path that works consistently, then 
I'll go back to the python to see about how to implement same.

I was getting 404's in the browser even, when I looked at it earlier.

I'll keep working on it, but not too hard, so as to not get on their radar in 
any unwanted sort of way.  ;-)

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 22, 2019, at 2:04 PM, Jarom Hatch  wrote:
> 
> Interesting, using curl sometimes I can it fine, but wunderfixer is always 
> getting a 403 Forbidden, as if it is actively being blocked...  When it 
> doesn't work in curl I get `HTTP/1.1 404 Not Found` and when it does work I 
> get `HTTP/1.1 200 OK`.  Curl never gets a 403 error.
> 
>> On Wednesday, May 22, 2019 at 11:48:08 AM UTC-6, Jarom Hatch wrote:
>> I was able to get it to work twice in my web browser, but as you said, it is 
>> sporadic.  I don't ever recall them using Akamai before so that may very 
>> well be a contributing factor.
>> 
>> I wonder if we can find out the origin address and see what happens if we 
>> can bypass Akamai...
>> 
>>> On Wednesday, May 22, 2019 at 7:35:18 AM UTC-6, Leon Shaner wrote:
>>> For one thing, the URL of this form:
>>> 
>>> http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1
>>> 
>>> Is now redirecting to one using HTTPS:
>>> 
>>> https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1
>>> 
>>> Also, the redirect itself takes an excruciatingly long time.
>>> So I just changed the URL to https directly...
>>> 
>>> The first time I tried any of the above using CURL this morning it worked, 
>>> but then after that I started getting:
>>> 
>>> An error occurred while processing your request.
>>> Reference #30.6f451160.1558531514.16ced4f6
>>> 
>>> It looks as if they've put some kind of Akamai proxy in the middle, which 
>>> is fine for static content, but not so fine for a query of this nature.  
>>> Strange that it worked for me the very first time.  It's almost as if the 
>>> Akamai "farm" has lost some "state" information and not all nodes have the 
>>> same content, so if you get stuck going through a bad node you get a bogus 
>>> response.
>>> 
>>> Attached is a transcript of a failed attempt.  I put SOMESTATION there only 
>>> after the fact.  The actual query was for my actual station, which used to 
>>> work.
>>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/07ac6f86-ae4d-4854-8398-ce4ab8d846c1%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/3260F8E7-37D4-4737-999B-97522E324370%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Jarom Hatch]

Interesting, using curl sometimes I can it fine, but wunderfixer is always 
getting a 403 Forbidden, as if it is actively being blocked...  When it 
doesn't work in curl I get `HTTP/1.1 404 Not Found` and when it does work I 
get `HTTP/1.1 200 OK`.  Curl never gets a 403 error.

On Wednesday, May 22, 2019 at 11:48:08 AM UTC-6, Jarom Hatch wrote:
>
> I was able to get it to work twice in my web browser, but as you said, it 
> is sporadic.  I don't ever recall them using Akamai before so that may very 
> well be a contributing factor.
>
> I wonder if we can find out the origin address and see what happens if we 
> can bypass Akamai...
>
> On Wednesday, May 22, 2019 at 7:35:18 AM UTC-6, Leon Shaner wrote:
>>
>> For one thing, the URL of this form:
>>
>>
>> http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1
>>
>> Is now redirecting to one using HTTPS:
>>
>>
>> https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1
>>
>> Also, the redirect itself takes an excruciatingly long time.
>> So I just changed the URL to https directly...
>>
>> The first time I tried any of the above using CURL this morning it 
>> worked, but then after that I started getting:
>>
>> An error occurred while processing your request.
>>
>> Reference #30.6f451160.1558531514.16ced4f6
>> It looks as if they've put some kind of Akamai proxy in the middle, which 
>> is fine for static content, but not so fine for a query of this nature. 
>>  Strange that it worked for me the very first time.  It's almost as if the 
>> Akamai "farm" has lost some "state" information and not all nodes have the 
>> same content, so if you get stuck going through a bad node you get a bogus 
>> response.
>>
>> Attached is a transcript of a failed attempt.  I put SOMESTATION there 
>> only after the fact.  The actual query was for my actual station, which 
>> used to work.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/07ac6f86-ae4d-4854-8398-ce4ab8d846c1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Jarom Hatch]

I was able to get it to work twice in my web browser, but as you said, it 
is sporadic.  I don't ever recall them using Akamai before so that may very 
well be a contributing factor.

I wonder if we can find out the origin address and see what happens if we 
can bypass Akamai...

On Wednesday, May 22, 2019 at 7:35:18 AM UTC-6, Leon Shaner wrote:
>
> For one thing, the URL of this form:
>
>
> http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1
>
> Is now redirecting to one using HTTPS:
>
>
> https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1
>
> Also, the redirect itself takes an excruciatingly long time.
> So I just changed the URL to https directly...
>
> The first time I tried any of the above using CURL this morning it worked, 
> but then after that I started getting:
>
> An error occurred while processing your request.
>
> Reference #30.6f451160.1558531514.16ced4f6
> It looks as if they've put some kind of Akamai proxy in the middle, which 
> is fine for static content, but not so fine for a query of this nature. 
>  Strange that it worked for me the very first time.  It's almost as if the 
> Akamai "farm" has lost some "state" information and not all nodes have the 
> same content, so if you get stuck going through a bad node you get a bogus 
> response.
>
> Attached is a transcript of a failed attempt.  I put SOMESTATION there 
> only after the fact.  The actual query was for my actual station, which 
> used to work.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/0089a003-7709-4d4a-82c5-8cb54cc2dc1a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

For one thing, the URL of this form:http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1Is now redirecting to one using HTTPS:https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1Also, the redirect itself takes an excruciatingly long time.So I just changed the URL to https directly...The first time I tried any of the above using CURL this morning it worked, but then after that I started getting:An error occurred while processing your request.Reference #30.6f451160.1558531514.16ced4f6It looks as if they've put some kind of Akamai proxy in the middle, which is fine for static content, but not so fine for a query of this nature.  Strange that it worked for me the very first time.  It's almost as if the Akamai "farm" has lost some "state" information and not all nodes have the same content, so if you get stuck going through a bad node you get a bogus response.Attached is a transcript of a failed attempt.  I put SOMESTATION there only after the fact.  The actual query was for my actual station, which used to work.



-- 
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/D629ECBA-956A-48D1-97F6-662D8A4EE70E%40isylum.org.
For more options, visit https://groups.google.com/d/optout.
curl -L -v 
"http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1";
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0* 
  Trying 184.27.188.101...
* TCP_NODELAY set
* Connected to www.wunderground.com (184.27.188.101) port 80 (#0)
> GET 
> /weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1
>  HTTP/1.1
> Host: www.wunderground.com
> User-Agent: curl/7.52.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Server: AkamaiGHost
< Content-Length: 0
< Location: 
https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1
< Cache-Control: max-age=0
< Expires: Wed, 22 May 2019 13:29:03 GMT
< Date: Wed, 22 May 2019 13:29:03 GMT
< Connection: keep-alive
< Set-Cookie: speedpin=4G; expires=Wed, 22-May-2019 13:59:03 GMT; path=/; 
domain=.wunderground.com; secure
< Set-Cookie: 
ci=TWC-Connection-Speed=4G&TWC-Locale-Group=US&TWC-Device-Class=desktop&X-Origin-Hint=dna&TWC-Network-Type=wifi&TWC-GeoIP-Country=US&TWC-GeoIP-Lat=42.3055&TWC-GeoIP-Long=-83.1724&Akamai-Connection-Speed=1000+&TWC-Privacy=exempt;
 path=/; domain=.wunderground.com; secure
< Property-id: drupal-prod
< X-RequestSource: AkamaiProdProxy
< TWC-Privacy: exempt
< TWC-GeoIP-LatLong: 42.3055,-83.1724
< TWC-GeoIP-Country: US
< TWC-Device-Class: desktop
< TWC-Locale-Group: US
< TWC-Connection-Speed: 4G
< X-RequestSource: AkamaiDefaultDNA
< 
* Curl_http_done: called premature == 0

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
* Connection #0 to host www.wunderground.com left intact
* Issue another request to this URL: 
'https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION&month=5&day=22&year=2019&format=1'
*   Trying 184.27.188.101...
* TCP_NODELAY set
* Connected to www.wunderground.com (184.27.188.101) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [102 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2527 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [148 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=Georgia; L=Atlanta; O=TWC Product and Technology, LLC; 
OU=Digital; CN=www.weather.com
*  start date: Sep 25 00:00:00 2018 GMT
*  expire date: Nov 24 12:00:00 2019 GMT
*  subjectAltName: host "www

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[gjr80]

Hmm, I notice that WXDailyHistory is working as usual now (well for me 
anyway) but wunderfixer still gives a 403 error.

Gary

On Wednesday, 22 May 2019 13:02:24 UTC+10, gjr80 wrote:
>
> Wunderfixer (as it stands now) relies on WUs WXDailyHistory.asp page. If 
> WXDailyHistory doesn't work then wunderfixer will not work. Period.
>
> I guess depending on where you sit on the conspiracy theory scale will 
> determine whether you believe the current issues with WXDailyHistory are 
> short term glitches or a silent closure of the service. There are a few 
> threads on wxforum.net that touch on this issue, and of course many 
> others that touch on the WU issues over the last not so long period. There 
> seems to have been a few posts from WU staff indicating that 'all the old 
> stuff is very fragile', so that could mean short term glitch, but on the 
> other hand WU has been pushing folks towards their 'new' API which could 
> mean the end for WXDailyHistory.
>
> The 'new' WU API that is available to PWS owners/uploaders does have a '1 
> day rapid history request' that provides what seems like the same sort of 
> info as WXDailyHistory, albeit in JSON format and with a whole pile more 
> cruft, well at least as far as wunderfixer is concerned. So in theory there 
> is no reason wunderfixer could not be reworked to utilise the '1 day rapid 
> history request' data. Of note to use the new API you need to have a (new) 
> API key, whereas WXDailyHistory/wunderfixer does not require an API key. 
> One shortcoming I did notice with the '1 day rapid history request' is 
> there appears to be no way to obtain a given days data; you get the current 
> day only, whereas with WXDailyHistory you can choose the date required, so 
> that would mean no more historical wunderfixing.
>
> In any case, those who may consider WU as a de facto backup may want to 
> start brushing up on their bash/MySQL skills and implement a robust local 
> backup arrangement (though on WUs form you should have done that ages ago).
>
> Gary
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/fac9e205-aca1-4376-92d6-b1ddcb889d19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[gjr80]

Wunderfixer (as it stands now) relies on WUs WXDailyHistory.asp page. If 
WXDailyHistory doesn't work then wunderfixer will not work. Period.

I guess depending on where you sit on the conspiracy theory scale will 
determine whether you believe the current issues with WXDailyHistory are 
short term glitches or a silent closure of the service. There are a few 
threads on wxforum.net that touch on this issue, and of course many others 
that touch on the WU issues over the last not so long period. There seems 
to have been a few posts from WU staff indicating that 'all the old stuff 
is very fragile', so that could mean short term glitch, but on the other 
hand WU has been pushing folks towards their 'new' API which could mean the 
end for WXDailyHistory.

The 'new' WU API that is available to PWS owners/uploaders does have a '1 
day rapid history request' that provides what seems like the same sort of 
info as WXDailyHistory, albeit in JSON format and with a whole pile more 
cruft, well at least as far as wunderfixer is concerned. So in theory there 
is no reason wunderfixer could not be reworked to utilise the '1 day rapid 
history request' data. Of note to use the new API you need to have a (new) 
API key, whereas WXDailyHistory/wunderfixer does not require an API key. 
One shortcoming I did notice with the '1 day rapid history request' is 
there appears to be no way to obtain a given days data; you get the current 
day only, whereas with WXDailyHistory you can choose the date required, so 
that would mean no more historical wunderfixing.

In any case, those who may consider WU as a de facto backup may want to 
start brushing up on their bash/MySQL skills and implement a robust local 
backup arrangement (though on WUs form you should have done that ages ago).

Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/eedc53a5-c2e8-4e96-8574-9f0cf152a480%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

FWIW, I had the odd 404 the other day, but then hours later it worked.
They may be just moving infrastructure around and not getting all the load 
balancer rules correct on the first go. Hope that's all it is...

Could it be that we need to request an API key and use that as a password, 
instead?

Here is what I see:

/usr/share/weewx/log/weewx.log.1:Thu 16 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Thu 16 May 13:17:01 EDT 2019 Reason: HTTP 
Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Thu 16 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Thu 16 May 13:17:01 EDT 2019 Reason: HTTP 
Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Fri 17 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Fri 17 May 13:17:01 EDT 2019 Reason: HTTP 
Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Fri 17 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Fri 17 May 13:17:01 EDT 2019 Reason: HTTP 
Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Sat 18 May 03:02:03 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Sat 18 May 03:02:03 EDT 2019 Reason: HTTP 
Error 404: Not Found
/usr/share/weewx/log/weewx.log:Mon 20 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Mon 20 May 13:17:01 EDT 2019 Reason: HTTP Error 
403: Forbidden
/usr/share/weewx/log/weewx.log:Mon 20 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Mon 20 May 13:17:01 EDT 2019 Reason: HTTP Error 
403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 01:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 01:17:01 EDT 2019 Reason: HTTP Error 
403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 01:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 01:17:01 EDT 2019 Reason: HTTP Error 
403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 13:17:01 EDT 2019 Reason: HTTP Error 
403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 13:17:01 EDT 2019 Reason: HTTP Error 
403: Forbidden


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 21, 2019, at 4:33 PM, Thomas Keffer  wrote:
> 
> It looks like the WU completely changed their API without warning. For 
> example, to access my station, it used to be
> https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KORHOODR3
> but now it's
> 
> https://www.wunderground.com/dashboard/pws/KORHOODR3
> You used to be able to download a day's worth of data, but now I'm not sure 
> how to do that without screen scraping. Wunderfixer may no longer be possible 
> to do, at least not accurately
> 
> -tk
> 
>> On Tue, May 21, 2019 at 12:46 PM Jarom Hatch  wrote:
>> I'm getting 403.  I entered an issue but upon looking further it appears to 
>> me that wunderground has either broken that url or intentionally deactivated 
>> it.  Hoping it isn't dead forever.
>> 
>>> On Tuesday, May 21, 2019 at 12:09:14 PM UTC-6, tds wrote:
>>> For the last three days, my daily cron job of wunderfixer have returned 
>>> "503 Unavaialbe" (May 18), "404 Not Found" (May 19) and "403 Forbidden" 
>>> (May 20).
>>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/54718d99-b68d-4f34-8213-4786a2578265%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/CAPq0zEDFjY2C%3DUt8Tnnx00gZE189-w-mJ%2BcwAaU%2BiCjOEZ5vsQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Jarom Hatch]

Just tried posting a question to WU using their email form.  Got this 
error:  

Validation failed: weather_underground is invalid.

Looks like they're a mess right now.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/013ac4c7-924c-4747-a29a-d02b3fbd5784%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Thomas Keffer]

It looks like the WU completely changed their API without warning. For
example, to access my station, it used to be

https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KORHOODR3

but now it's

https://www.wunderground.com/dashboard/pws/KORHOODR3

You used to be able to download a day's worth of data, but now I'm not sure
how to do that without screen scraping. Wunderfixer may no longer be
possible to do, at least not accurately


-tk

On Tue, May 21, 2019 at 12:46 PM Jarom Hatch  wrote:

> I'm getting 403.  I entered an issue but upon looking further it appears
> to me that wunderground has either broken that url or intentionally
> deactivated it.  Hoping it isn't dead forever.
>
> On Tuesday, May 21, 2019 at 12:09:14 PM UTC-6, tds wrote:
>>
>> For the last three days, my daily cron job of wunderfixer have returned
>> "503 Unavaialbe" (May 18), "404 Not Found" (May 19) and "403 Forbidden"
>> (May 20).
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/54718d99-b68d-4f34-8213-4786a2578265%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEDFjY2C%3DUt8Tnnx00gZE189-w-mJ%2BcwAaU%2BiCjOEZ5vsQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.