I run wunderfixer twice per day - midnight and midday. I occasionally have missed records, but mostly it seems fine. I do NOT use rapid fire and I upload every 5 minutes to WU. Past experience tells me that WU appears to try and retain 5 minute intervals between readings at 00,05,10,15 etc etc ... readings submitted at 'odd' intervals are likely to 'get lost' and ultimately a record on the 5 minute boundary will appear.
Conclusion. do not overload WU. do not submit rapid fire and archive records - use one or the other. On Wednesday, 1 May 2019 18:07:06 UTC+3, Leon Shaner wrote: > > Hey, WeeWX'ers, > > I wonder how many folks are experiencing issues with WU timing out / not > responding / failing to accept station updates from weewx? > > I've resorted to running wunderfixer roughly every 10 minutes against > "today" and "yesterday" and it's almost always finding missing records > throughout "today" which after enough wunderfixer re-uploads, they do > eventually "stick." > > I can see from my "weewx_watchdog" logs that wunderfixer was timing out > almost every attempt to run against the "yesterday" date, but was usually > fine when running against the "today" date. > I'm guessing that WU may have some optimizations in place to respond > quickly to "current" day queries, and maybe they have intentionally > allocated less "capacity" to historical queries. That might explain the > timeouts I was seeing mainly when querying the prior day. > > I found that increasing the wunderfixer timeout from 10 seconds to 20 > seconds was helpful in working around the WU connection issues. There is > an updated wunderfixer in the that accepts a --timeout option, over here. > Thanks, Tom! =D > > https://github.com/weewx/weewx/blob/master/bin/wunderfixer > > > Now, because wunderfixer is pretty much always finding missing records, I > am wondering of weewx itself is falling victim to the WU "capacity issues" > during the normal uploading of records? > As in could the records be missing on the WU side due to similar timeouts > that wunderfixer is experiencing? > > I have debug = 1, but there are no issues being reported, so do I need a > higher debug, such as debut = 3? > > Could it be that the connections are going through but WU is losing the > data, anyway? > > And/or could it be that running wunderfixer every 10 minutes isn't > allowing WU enough time to process the data, such that is is available by > the next query, 10 minutes later? > > > Here are some logs of what I am seeing re: the trend of missing WU data. > You can see the "overlap" in that multiple runs of wunderfixer 10 minutes > apart are finding and re-uploading the exact same missing records until > eventually they do "stick" in that they are not shown missing / not > re-uploaded. > > In the below, you can see that a record from 1556709420 did get posted > properly, because it was not mentioned again 10 minutes later. However, > there are three other records in *bold*, which did not "stick" between > those wunderfixer runs, 10 minutes apart. And then 10 minutes later, two > more overlapping re-uploads, which didn't "stick." > > > Wed 1 May 10:15:11 EDT 2019 Weather Underground Station: KMIDEARB5 > Wed 1 May 10:15:11 EDT 2019 Date to check: 2019-04-30 > Wed 1 May 10:15:11 EDT 2019 Number of archive records: 1438 > Wed 1 May 10:15:11 EDT 2019 Number of WU records: 381 > Wed 1 May 10:15:11 EDT 2019 Number of missing records: 0 > Wed 1 May 10:25:01 EDT 2019 Using configuration file > /etc/weewx/weewx.conf. > Wed 1 May 10:25:01 EDT 2019 Using database binding 'wx_binding', which is > bound to database 'archive_sqlite' > Wed 1 May 10:25:01 EDT 2019 Weather Underground Station: KMIDEARB5 > Wed 1 May 10:25:01 EDT 2019 Date to check: 2019-05-01 > Wed 1 May 10:25:01 EDT 2019 Number of archive records: 624 > Wed 1 May 10:25:01 EDT 2019 Number of WU records: 144 > Wed 1 May 10:25:01 EDT 2019 Number of missing records: 5 > Wed 1 May 10:25:01 EDT 2019 > Wed 1 May 10:25:01 EDT 2019 Missing records: > Wed 1 May 10:25:01 EDT 2019 2019-05-01 07:17:00 EDT (1556709420); > 29.297"; 47.7F; 93%; 4.7 mph; 98 deg; 7.6 mph gust; 45.7F; 0.00" rain > ...published. > Wed 1 May 10:25:01 EDT 2019 2019-05-01 09:57:00 EDT (*1556719020*); > 29.226"; 50.4F; 93%; 2.9 mph; 111 deg; 6.9 mph gust; 48.4F; 0.00" rain > ...published. > Wed 1 May 10:25:01 EDT 2019 2019-05-01 10:17:00 EDT (*1556720220*); > 29.223"; 52.0F; 92%; 2.0 mph; 136 deg; 3.6 mph gust; 49.9F; 0.00" rain > ...published. > Wed 1 May 10:25:01 EDT 2019 2019-05-01 10:23:00 EDT (1556720580); > 29.214"; 52.4F; 92%; 2.5 mph; 115 deg; 4.3 mph gust; 50.1F; 0.00" rain > ...published. > Wed 1 May 10:25:01 EDT 2019 2019-05-01 10:24:00 EDT (1556720640); > 29.214"; 52.5F; 92%; 2.5 mph; 115 deg; 5.4 mph gust; 50.3F; 0.00" rain > ...published. > Wed 1 May 10:25:14 EDT 2019 Using configuration file > /etc/weewx/weewx.conf. > Wed 1 May 10:25:14 EDT 2019 Using database binding 'wx_binding', which is > bound to database 'archive_sqlite' > Wed 1 May 10:25:14 EDT 2019 Weather Underground Station: KMIDEARB5 > Wed 1 May 10:25:14 EDT 2019 Date to check: 2019-04-30 > Wed 1 May 10:25:14 EDT 2019 Number of archive records: 1438 > Wed 1 May 10:25:14 EDT 2019 Number of WU records: 381 > Wed 1 May 10:25:14 EDT 2019 Number of missing records: 0 > Wed 1 May 10:35:01 EDT 2019 Using configuration file > /etc/weewx/weewx.conf. > Wed 1 May 10:35:01 EDT 2019 Using database binding 'wx_binding', which is > bound to database 'archive_sqlite' > Wed 1 May 10:35:01 EDT 2019 Weather Underground Station: KMIDEARB5 > Wed 1 May 10:35:01 EDT 2019 Date to check: 2019-05-01 > Wed 1 May 10:35:01 EDT 2019 Number of archive records: 634 > Wed 1 May 10:35:01 EDT 2019 Number of WU records: 147 > Wed 1 May 10:35:01 EDT 2019 Number of missing records: 6 > Wed 1 May 10:35:01 EDT 2019 > Wed 1 May 10:35:01 EDT 2019 Missing records: > Wed 1 May 10:35:01 EDT 2019 2019-05-01 09:57:00 EDT (*1556719020*); > 29.226"; 50.4F; 93%; 2.9 mph; 111 deg; 6.9 mph gust; 48.4F; 0.00" rain > ...published. > Wed 1 May 10:35:01 EDT 2019 2019-05-01 10:17:00 EDT (*1556720220*); > 29.223"; 52.0F; 92%; 2.0 mph; 136 deg; 3.6 mph gust; 49.9F; 0.00" rain > ...published. > Wed 1 May 10:35:01 EDT 2019 2019-05-01 10:27:00 EDT (*1556720820*); > 29.214"; 52.9F; 92%; 2.5 mph; 115 deg; 6.5 mph gust; 50.6F; 0.00" rain > ...published. > Wed 1 May 10:35:01 EDT 2019 2019-05-01 10:32:00 EDT (*1556721120*); > 29.214"; 53.2F; 91%; 2.5 mph; 113 deg; 2.5 mph gust; 50.7F; 0.00" rain > ...published. > Wed 1 May 10:35:01 EDT 2019 2019-05-01 10:33:00 EDT (1556721180); > 29.214"; 53.2F; 91%; 2.5 mph; 113 deg; 4.3 mph gust; 50.7F; 0.00" rain > ...published. > Wed 1 May 10:35:01 EDT 2019 2019-05-01 10:34:00 EDT (1556721240); > 29.207"; 53.4F; 91%; 2.5 mph; 113 deg; 3.6 mph gust; 50.9F; 0.00" rain > ...published. > > And again 10 minutes later: > > Wed 1 May 10:45:01 EDT 2019 Weather Underground Station: KMIDEARB5 > Wed 1 May 10:45:01 EDT 2019 Date to check: 2019-05-01 > Wed 1 May 10:45:01 EDT 2019 Number of archive records: 644 > Wed 1 May 10:45:01 EDT 2019 Number of WU records: 149 > Wed 1 May 10:45:01 EDT 2019 Number of missing records: 6 > Wed 1 May 10:45:01 EDT 2019 > Wed 1 May 10:45:01 EDT 2019 Missing records: > Wed 1 May 10:45:01 EDT 2019 2019-05-01 09:57:00 EDT (*1556719020*); > 29.226"; 50.4F; 93%; 2.9 mph; 111 deg; 6.9 mph gust; 48.4F; 0.00" rain > ...published. > Wed 1 May 10:45:01 EDT 2019 2019-05-01 10:17:00 EDT (*1556720220*); > 29.223"; 52.0F; 92%; 2.0 mph; 136 deg; 3.6 mph gust; 49.9F; 0.00" rain > ...published. > Wed 1 May 10:45:01 EDT 2019 2019-05-01 10:27:00 EDT (*1556720820*); > 29.214"; 52.9F; 92%; 2.5 mph; 115 deg; 6.5 mph gust; 50.6F; 0.00" rain > ...published. > Wed 1 May 10:45:01 EDT 2019 2019-05-01 10:32:00 EDT (*1556721120*); > 29.214"; 53.2F; 91%; 2.5 mph; 113 deg; 2.5 mph gust; 50.7F; 0.00" rain > ...published. > Wed 1 May 10:45:01 EDT 2019 2019-05-01 10:43:00 EDT (1556721780); > 29.202"; 54.1F; 91%; 2.0 mph; 111 deg; 6.5 mph gust; 51.4F; 0.00" rain > ...published. > Wed 1 May 10:45:01 EDT 2019 2019-05-01 10:44:00 EDT (1556721840); > 29.202"; 54.1F; 90%; 2.0 mph; 111 deg; 2.9 mph gust; 51.3F; 0.00" rain > ...published. > > It's like a "rolling window" of missing records that eventually do catch > up, but seemingly that is only because I am being persistent about running > wunderfixer. > > I could try running it only every 20 minutes to see if results change. > Let me think about that some more. :-/ > > Meanwhile, about the connection issues, from when wunderfixer timeout was > 10 seconds, vs. after changing it to 20 seconds. > The timeouts still occur, but less frequently > > ### timeouts almost every time I run wunderfixer 10 minutes apart, for > prior day > Tue 30 Apr 09:55:01 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 10:45:02 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 10:55:02 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 11:05:01 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 11:15:02 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 12:05:01 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 12:35:02 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 12:45:01 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 12:55:02 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 13:05:01 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 13:15:15 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 13:35:01 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 13:55:01 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 13:55:15 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 14:35:12 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 14:45:01 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 14:55:02 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 15:15:01 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 15:55:19 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 16:05:13 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 16:15:19 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 16:25:14 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 16:35:19 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 16:45:02 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 16:45:15 EDT 2019 Could not get Weather Underground data. > Exiting. > ### timeout changed from 10 seconds to 20 seconds; far fewer timeouts > occurring > Tue 30 Apr 21:55:23 EDT 2019 Could not get Weather Underground data. > Exiting. > Tue 30 Apr 23:15:33 EDT 2019 Could not get Weather Underground data. > Exiting. > Wed 1 May 01:15:20 EDT 2019 Could not get Weather Underground data. > Exiting. > Wed 1 May 01:25:14 EDT 2019 Could not get Weather Underground data. > Exiting. > Wed 1 May 01:35:15 EDT 2019 Could not get Weather Underground data. > Exiting. > ### no timeouts seen since that time, so far, today > > Regards, > \Leon > -- > Leon Shaner :: Dearborn, Michigan (iPad Pro) > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
