Re: [weewx-user] Re: System has stopped reporting everywhere

[Peter Dougherty]

Just a quick reply to say thanks, everything's working again, and the 
wunderfixer option is running OK. I have no idea what the end part of that 
command means, but I can see the result of the daily run in the log so 
that's all that matters. Thanks again for your assistance.

On Wednesday, May 24, 2017 at 10:26:40 PM UTC-4, gjr80 wrote:
>
> wunderfixer is included with weeWX, its location will be depend on how 
> weeWX was installed, it will most likely be in the /home/weewx/bin 
> directory but it could be in /usr/share/weewx if you installed from a 
> .deb or .rpm package. To run wunderfixer:
>
> $ /home/weewx/bin/wunderfixer --test
>
> if this generates an error it could be because the currently logged in 
> user does not have permission to do some of the things wunderfixer does 
> so in that case try running as root using:
>
> $ sudo /home/weewx/bin/wunderfixer --test
>
> the --test option gets wunderfixer to check WU for any missing records 
> for the current day and give you a summary report of what it found and what 
> it would do ('fix') but not actually do anything. By default wunderfixer 
> checks the current day. The --date option lets you specify another date 
> to check. So to check 17 May 2017 you would use:
>
> $  sudo /home/weewx/bin/wunderfixer --test --date=2017-05-17
>
> If you have 6 or 7 days to 'fix' then you will need to run wunderfixer 6 
> or 7 times, one for each day. To actually 'fix' a day just remove the 
> --test option. All of the other options can probably be left alone, 
> though the --query option can be helpful if you think your weeWX data may 
> be suspect, as you will be asked to confirm (y/n) before each WU record is 
> updated (if you have a whole day to 'fix' at 5 minute intervals that is 288 
> confirmations). Be aware that once data is published to WU you cannot 
> change it other than deleting it. So if you have any basis for thinking you 
> weeWX data is suspect, it is best to fix your weeWX data before using 
> wunderfixer.
>
> You can safely run wunderfixer for a give day as many times as you want, 
> at worst WU will ignore any attempted 'fixes' for data already recorded. I 
> always find it funny that if you run wunderfixer 5 times for a given day 
> the chances are you will get 5 different results in terms on number of 
> missing records - just the vagaries of WU.
>
> Having wunderfixer run as a cron job is easily setup. Best to have it run 
> under the root user cron so open the root user crontab using:
>
> $ sudo crontab -e
>
> Go to the bottom of the file and type something like (assuming wunderfixer 
> is in /home/weewx/bin):
>
> # Run wunderfixer at 1158pm daily
> 58 23 * * * /home/weewx/bin/wunderfixer --log weewx > /dev/null 2>&1
>
> This will run wunderfixer at 11:58pm each night, you can change it to 
> suit your requriements, but on a 5 minute interval system 11:58pm captures 
> all records for that particular day. Save the file (most likely Control-O 
> followed by enter then Control-X though depends on your editor). Note that 
> it is a good idea to run wunderfixer at least once from the command line 
> without the --test option before setting up the cron job, that way you 
> know that the WU credentials in your weewx.conf work fine with wunderfixer
> .
>
> Gary
>
> On Thursday, 25 May 2017 11:28:56 UTC+10, Peter Dougherty wrote:
>>
>> Hi Gary,
>> I've read through that link you sent until my eyes bled, and I just don't 
>> understand how to use it. Like I said up top, Linux and I aren't friends, 
>> we just tolerate each other's existence and call it a draw. So where do I 
>> go to get a copy of wunderfixer, how do I install it, configure it to look 
>> for every single piece of missing data and then run it? There are about 
>> 1,800 updates that never got sent to WU in the 6+ days it was down.
>>
>> On Tuesday, May 23, 2017 at 9:44:05 PM UTC-4, gjr80 wrote:
>>>
>>> Can't say much other than 'yes it is not posting to WU' without seeing 
>>> more of the log ie from weeWX startup. You can use wunderfixer 
>>>  to re-post to 
>>> WU. In fact it is a good idea to set up wunderfixer to run by cron on a 
>>> daily basis, I run it at 23:58 each night. WU is by no means perfect and 
>>> despite seeing 288 posts to WU each day in my logs some just don't make it.
>>>
>>> As to your previous questions. Twitter should not be that hard to set to 
>>> work, I would be getting everything else back and stable before worrying 
>>> about twitter. The error in the log indicates that python cannot find the 
>>> requests_oauthlib module. This is not included in the standard python 
>>> install, did you ever install it separately? Re-installing it would be a 
>>> good starting point. You may also find more info by searching the forums 
>>> for 'twitter' or maybe by looking at any comments at the start of the 
>>> bin/user/twitter.py 
>>> file.
>>>
>>> I think Tom covered the downloading of data from the 

Re: [weewx-user] Re: System has stopped reporting everywhere

[gjr80]

wunderfixer is included with weeWX, its location will be depend on how 
weeWX was installed, it will most likely be in the /home/weewx/bin 
directory but it could be in /usr/share/weewx if you installed from a .deb 
or .rpm package. To run wunderfixer:

$ /home/weewx/bin/wunderfixer --test

if this generates an error it could be because the currently logged in user 
does not have permission to do some of the things wunderfixer does so in 
that case try running as root using:

$ sudo /home/weewx/bin/wunderfixer --test

the --test option gets wunderfixer to check WU for any missing records for 
the current day and give you a summary report of what it found and what it 
would do ('fix') but not actually do anything. By default wunderfixer 
checks the current day. The --date option lets you specify another date to 
check. So to check 17 May 2017 you would use:

$  sudo /home/weewx/bin/wunderfixer --test --date=2017-05-17

If you have 6 or 7 days to 'fix' then you will need to run wunderfixer 6 or 
7 times, one for each day. To actually 'fix' a day just remove the --test 
option. All of the other options can probably be left alone, though the 
--query option can be helpful if you think your weeWX data may be suspect, 
as you will be asked to confirm (y/n) before each WU record is updated (if 
you have a whole day to 'fix' at 5 minute intervals that is 288 
confirmations). Be aware that once data is published to WU you cannot 
change it other than deleting it. So if you have any basis for thinking you 
weeWX data is suspect, it is best to fix your weeWX data before using 
wunderfixer.

You can safely run wunderfixer for a give day as many times as you want, at 
worst WU will ignore any attempted 'fixes' for data already recorded. I 
always find it funny that if you run wunderfixer 5 times for a given day 
the chances are you will get 5 different results in terms on number of 
missing records - just the vagaries of WU.

Having wunderfixer run as a cron job is easily setup. Best to have it run 
under the root user cron so open the root user crontab using:

$ sudo crontab -e

Go to the bottom of the file and type something like (assuming wunderfixer 
is in /home/weewx/bin):

# Run wunderfixer at 1158pm daily
58 23 * * * /home/weewx/bin/wunderfixer --log weewx > /dev/null 2>&1

This will run wunderfixer at 11:58pm each night, you can change it to suit 
your requriements, but on a 5 minute interval system 11:58pm captures all 
records for that particular day. Save the file (most likely Control-O 
followed by enter then Control-X though depends on your editor). Note that 
it is a good idea to run wunderfixer at least once from the command line 
without the --test option before setting up the cron job, that way you know 
that the WU credentials in your weewx.conf work fine with wunderfixer.

Gary

On Thursday, 25 May 2017 11:28:56 UTC+10, Peter Dougherty wrote:
>
> Hi Gary,
> I've read through that link you sent until my eyes bled, and I just don't 
> understand how to use it. Like I said up top, Linux and I aren't friends, 
> we just tolerate each other's existence and call it a draw. So where do I 
> go to get a copy of wunderfixer, how do I install it, configure it to look 
> for every single piece of missing data and then run it? There are about 
> 1,800 updates that never got sent to WU in the 6+ days it was down.
>
> On Tuesday, May 23, 2017 at 9:44:05 PM UTC-4, gjr80 wrote:
>>
>> Can't say much other than 'yes it is not posting to WU' without seeing 
>> more of the log ie from weeWX startup. You can use wunderfixer 
>>  to re-post to 
>> WU. In fact it is a good idea to set up wunderfixer to run by cron on a 
>> daily basis, I run it at 23:58 each night. WU is by no means perfect and 
>> despite seeing 288 posts to WU each day in my logs some just don't make it.
>>
>> As to your previous questions. Twitter should not be that hard to set to 
>> work, I would be getting everything else back and stable before worrying 
>> about twitter. The error in the log indicates that python cannot find the 
>> requests_oauthlib module. This is not included in the standard python 
>> install, did you ever install it separately? Re-installing it would be a 
>> good starting point. You may also find more info by searching the forums 
>> for 'twitter' or maybe by looking at any comments at the start of the 
>> bin/user/twitter.py 
>> file.
>>
>> I think Tom covered the downloading of data from the logger, another easy 
>> way to tell if the last 27 hours of data has been downloaded OK is to look 
>> at your daily plots. They default to 27 hours of data with no aggregation 
>> and any (extended) missing data will show up as one or more gaps. Week, 
>> month and year plots use aggregated data so gaps may not be apparent there.
>>
>> Gary
>>
>> On Wednesday, 24 May 2017 11:22:12 UTC+10, Peter Dougherty wrote:
>>>
>>> It's doesn't appear to be sending any of the missing data to 

Re: [weewx-user] Re: System has stopped reporting everywhere

[Peter Dougherty]

Hi Gary,
I've read through that link you sent until my eyes bled, and I just don't 
understand how to use it. Like I said up top, Linux and I aren't friends, 
we just tolerate each other's existence and call it a draw. So where do I 
go to get a copy of wunderfixer, how do I install it, configure it to look 
for every single piece of missing data and then run it? There are about 
1,800 updates that never got sent to WU in the 6+ days it was down.

On Tuesday, May 23, 2017 at 9:44:05 PM UTC-4, gjr80 wrote:
>
> Can't say much other than 'yes it is not posting to WU' without seeing 
> more of the log ie from weeWX startup. You can use wunderfixer 
>  to re-post to 
> WU. In fact it is a good idea to set up wunderfixer to run by cron on a 
> daily basis, I run it at 23:58 each night. WU is by no means perfect and 
> despite seeing 288 posts to WU each day in my logs some just don't make it.
>
> As to your previous questions. Twitter should not be that hard to set to 
> work, I would be getting everything else back and stable before worrying 
> about twitter. The error in the log indicates that python cannot find the 
> requests_oauthlib module. This is not included in the standard python 
> install, did you ever install it separately? Re-installing it would be a 
> good starting point. You may also find more info by searching the forums 
> for 'twitter' or maybe by looking at any comments at the start of the 
> bin/user/twitter.py 
> file.
>
> I think Tom covered the downloading of data from the logger, another easy 
> way to tell if the last 27 hours of data has been downloaded OK is to look 
> at your daily plots. They default to 27 hours of data with no aggregation 
> and any (extended) missing data will show up as one or more gaps. Week, 
> month and year plots use aggregated data so gaps may not be apparent there.
>
> Gary
>
> On Wednesday, 24 May 2017 11:22:12 UTC+10, Peter Dougherty wrote:
>>
>> It's doesn't appear to be sending any of the missing data to Weather 
>> Underground, but it *is* sending current data. The station went back 
>> online around 18:29 today (23 May) and had been offline starting around 
>> 05:00 on 17 May. Three hours after going back online I cannot find the 
>> missing data on the WU site.
>>
>> This is a log excerpt. It appears as if it's trying to send the backlog 
>> (there are hundreds of these entries in the log).
>>
>> .
>> May 23 18:34:32 wx weewx[1905]: manager: added record 2017-05-22 02:20:00 
>> EDT (1495434000) to database 'weewx.sdb'
>> May 23 18:34:32 wx weewx[1905]: manager: added record 2017-05-22 02:20:00 
>> EDT (1495434000) to daily summary in 'weewx.sdb'
>> May 23 18:34:32 wx weewx[1905]: restx: PWSWeather: Published record 
>> 2017-05-22 02:15:00 EDT (1495433700)
>> May 23 18:34:33 wx weewx[1905]: restx: CWOP: record 2017-05-22 02:20:00 
>> EDT (1495434000) is stale (144873 > 60).
>> May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:25:00 
>> EDT (1495434300) to database 'weewx.sdb'
>> May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:25:00 
>> EDT (1495434300) to daily summary in 'weewx.sdb'
>> May 23 18:34:33 wx weewx[1905]: restx: PWSWeather: Published record 
>> 2017-05-22 02:20:00 EDT (1495434000)
>> May 23 18:34:33 wx weewx[1905]: restx: CWOP: record 2017-05-22 02:25:00 
>> EDT (1495434300) is stale (144573 > 60).
>> May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:30:00 
>> EDT (1495434600) to database 'weewx.sdb'
>> May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:30:00 
>> EDT (1495434600) to daily summary in 'weewx.sdb'
>> May 23 18:34:33 wx weewx[1905]: restx: PWSWeather: Published record 
>> 2017-05-22 02:25:00 EDT (1495434300)
>> May 23 18:34:33 wx weewx[1905]: restx: CWOP: record 2017-05-22 02:30:00 
>> EDT (1495434600) is stale (144273 > 60).
>> May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:35:00 
>> EDT (1495434900) to database 'weewx.sdb'
>> May 23 18:34:33 wx weewx[1905]: restx: PWSWeather: Published record 
>> 2017-05-22 02:30:00 EDT (1495434600)
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: System has stopped reporting everywhere

[gjr80]

Can't say much other than 'yes it is not posting to WU' without seeing more 
of the log ie from weeWX startup. You can use wunderfixer 
 to re-post to WU. 
In fact it is a good idea to set up wunderfixer to run by cron on a daily 
basis, I run it at 23:58 each night. WU is by no means perfect and despite 
seeing 288 posts to WU each day in my logs some just don't make it.

As to your previous questions. Twitter should not be that hard to set to 
work, I would be getting everything else back and stable before worrying 
about twitter. The error in the log indicates that python cannot find the 
requests_oauthlib module. This is not included in the standard python 
install, did you ever install it separately? Re-installing it would be a 
good starting point. You may also find more info by searching the forums 
for 'twitter' or maybe by looking at any comments at the start of the 
bin/user/twitter.py 
file.

I think Tom covered the downloading of data from the logger, another easy 
way to tell if the last 27 hours of data has been downloaded OK is to look 
at your daily plots. They default to 27 hours of data with no aggregation 
and any (extended) missing data will show up as one or more gaps. Week, 
month and year plots use aggregated data so gaps may not be apparent there.

Gary

On Wednesday, 24 May 2017 11:22:12 UTC+10, Peter Dougherty wrote:
>
> It's doesn't appear to be sending any of the missing data to Weather 
> Underground, but it *is* sending current data. The station went back 
> online around 18:29 today (23 May) and had been offline starting around 
> 05:00 on 17 May. Three hours after going back online I cannot find the 
> missing data on the WU site.
>
> This is a log excerpt. It appears as if it's trying to send the backlog 
> (there are hundreds of these entries in the log).
>
> .
> May 23 18:34:32 wx weewx[1905]: manager: added record 2017-05-22 02:20:00 
> EDT (1495434000) to database 'weewx.sdb'
> May 23 18:34:32 wx weewx[1905]: manager: added record 2017-05-22 02:20:00 
> EDT (1495434000) to daily summary in 'weewx.sdb'
> May 23 18:34:32 wx weewx[1905]: restx: PWSWeather: Published record 
> 2017-05-22 02:15:00 EDT (1495433700)
> May 23 18:34:33 wx weewx[1905]: restx: CWOP: record 2017-05-22 02:20:00 
> EDT (1495434000) is stale (144873 > 60).
> May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:25:00 
> EDT (1495434300) to database 'weewx.sdb'
> May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:25:00 
> EDT (1495434300) to daily summary in 'weewx.sdb'
> May 23 18:34:33 wx weewx[1905]: restx: PWSWeather: Published record 
> 2017-05-22 02:20:00 EDT (1495434000)
> May 23 18:34:33 wx weewx[1905]: restx: CWOP: record 2017-05-22 02:25:00 
> EDT (1495434300) is stale (144573 > 60).
> May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:30:00 
> EDT (1495434600) to database 'weewx.sdb'
> May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:30:00 
> EDT (1495434600) to daily summary in 'weewx.sdb'
> May 23 18:34:33 wx weewx[1905]: restx: PWSWeather: Published record 
> 2017-05-22 02:25:00 EDT (1495434300)
> May 23 18:34:33 wx weewx[1905]: restx: CWOP: record 2017-05-22 02:30:00 
> EDT (1495434600) is stale (144273 > 60).
> May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:35:00 
> EDT (1495434900) to database 'weewx.sdb'
> May 23 18:34:33 wx weewx[1905]: restx: PWSWeather: Published record 
> 2017-05-22 02:30:00 EDT (1495434600)
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: System has stopped reporting everywhere

[Peter Dougherty]

It's doesn't appear to be sending any of the missing data to Weather 
Underground, but it *is* sending current data. The station went back online 
around 18:29 today (23 May) and had been offline starting around 05:00 on 
17 May. Three hours after going back online I cannot find the missing data 
on the WU site.

This is a log excerpt. It appears as if it's trying to send the backlog 
(there are hundreds of these entries in the log).

.
May 23 18:34:32 wx weewx[1905]: manager: added record 2017-05-22 02:20:00 
EDT (1495434000) to database 'weewx.sdb'
May 23 18:34:32 wx weewx[1905]: manager: added record 2017-05-22 02:20:00 
EDT (1495434000) to daily summary in 'weewx.sdb'
May 23 18:34:32 wx weewx[1905]: restx: PWSWeather: Published record 
2017-05-22 02:15:00 EDT (1495433700)
May 23 18:34:33 wx weewx[1905]: restx: CWOP: record 2017-05-22 02:20:00 EDT 
(1495434000) is stale (144873 > 60).
May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:25:00 
EDT (1495434300) to database 'weewx.sdb'
May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:25:00 
EDT (1495434300) to daily summary in 'weewx.sdb'
May 23 18:34:33 wx weewx[1905]: restx: PWSWeather: Published record 
2017-05-22 02:20:00 EDT (1495434000)
May 23 18:34:33 wx weewx[1905]: restx: CWOP: record 2017-05-22 02:25:00 EDT 
(1495434300) is stale (144573 > 60).
May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:30:00 
EDT (1495434600) to database 'weewx.sdb'
May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:30:00 
EDT (1495434600) to daily summary in 'weewx.sdb'
May 23 18:34:33 wx weewx[1905]: restx: PWSWeather: Published record 
2017-05-22 02:25:00 EDT (1495434300)
May 23 18:34:33 wx weewx[1905]: restx: CWOP: record 2017-05-22 02:30:00 EDT 
(1495434600) is stale (144273 > 60).
May 23 18:34:33 wx weewx[1905]: manager: added record 2017-05-22 02:35:00 
EDT (1495434900) to database 'weewx.sdb'
May 23 18:34:33 wx weewx[1905]: restx: PWSWeather: Published record 
2017-05-22 02:30:00 EDT (1495434600)

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: System has stopped reporting everywhere

[Thomas Keffer]

If the option record_generation is set to 'hardware' (the default), then
weewx will download all records off the data logger since the last record
it has in your database. As it does so, it sends entries to the WU. So, it
is always "up to date."

You can watch this happen by setting debug=1 in weewx.conf. The log will
look something like this:

May 23 15:40:16 nuc weewx[20710]: vantage: Getting archive packets since
2017-05-23 15:35:00 PDT (1495578900)
May 23 15:40:16 nuc weewx[20710]: vantage: gentle wake up of console
successful
May 23 15:40:16 nuc weewx[20710]: vantage: Retrieving 1 page(s); starting
index= 3
May 23 15:40:16 nuc weewx[20710]: manager: Added record 2017-05-23 15:40:00
PDT (1495579200) to database 'weewx.sdb'
May 23 15:40:16 nuc weewx[20710]: manager: Added record 2017-05-23 15:40:00
PDT (1495579200) to daily summary in 'weewx.sdb'
May 23 15:40:16 nuc weewx[20710]: vantage: DMPAFT complete: page timestamp
2017-05-14 18:20:00 PDT (1494811200) less than final timestamp 2
017-05-23 15:40:00 PDT (1495579200)
May 23 15:40:16 nuc weewx[20710]: vantage: Catch up complete.

In this example, the last entry in the database is 2017-05-23 15:35:00.
Weewx found a single new entry in the logger since that time: 2017-05-23
15:40:00.

And so on...

-tk


On Tue, May 23, 2017 at 5:10 PM, Peter Dougherty 
wrote:

> OK, done and that got things working again. I'm curious as to what broke.
> No changes were made since the end of February, and the system has been up
> and reporting continuously since then. My tech guru and I attempted to get
> the Twitter reporting function working (like I had with Cumulus software,
> under Windows) but it never would, and so we just ignored it.
>
> Is there any way to determine if the last 5 days' data has been
> successfully downloaded from the data logger on the Vantage Pro 2? And if
> so, is there any way that you know if to retroactively upload it to Weather
> Underground's PWS feature. Thanks in advance.
>
>
> On Tuesday, May 23, 2017 at 6:19:42 PM UTC-4, gjr80 wrote:
>>
>> Ok,disable Twitter in restful_services by commenting out:
>>
>> user.twitter.Twitter
>>
>> Save weewx.conf, stop then start weeWX and lets see what is in the 10
>> minute log then.
>>
>> Gary
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.