Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Joe Brockway]

Victory!

I had the --drop-daily first and then --rebuild-daily. The rebuilding of 
the the dailies took several minutes, but it appears the NOAA reports are 
correct now.

Thank you Gary. Thank you Andrew.


On Tuesday, December 5, 2017 at 2:27:27 AM UTC-7, gjr80 wrote:
>
> Yep, looks like the daily summaries only have data from 2nd December 
> onwards. Chances are a rebuild using wee_database will fix the problem, 
> refer to the Utilites Guide  
> http://weewx.com/docs/utilities.htm#Action_--rebuild-daily. Make sure you 
> stop weeWX first.
>
> Once the daily summaries are rebuilt you will need to delete the month and 
> year summaries to force a rebuild from scratch.
>
> Gary
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[gjr80]

Yep, looks like the daily summaries only have data from 2nd December onwards. 
Chances are a rebuild using wee_database will fix the problem, refer to the 
Utilites Guide  http://weewx.com/docs/utilities.htm#Action_--rebuild-daily. 
Make sure you stop weeWX first.

Once the daily summaries are rebuilt you will need to delete the month and year 
summaries to force a rebuild from scratch.

Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Andrew Milner]

Gary
All the plots - day, week, month, year appear to show data from the 
database - I assume these plots are generated from the archive table, and 
the yearly plot generated at midnight on 5th December shows plausible data 
for all of 2017 - so the data exists somewhere!!

The high/lows, as you rightly say, all seem to be from December 2017, and 
the NOAA reports are blank - I assume this data comes from the summary 
tables

Leading to my conclusion that the summary tables need recreating from the 
archive data - but no doubt your queries will confirm/reject this theory.





On Tuesday, 5 December 2017 09:57:12 UTC+2, gjr80 wrote:

> Ok, must have been some port filtering on the internet I was using. I 
> suspect that the only data in your database is 2nd December onwards only, 
> look at all your month and year highs/lows, they are identical. December 
> 2017 must have been a memorable month, or you only have data for December 
> 2017. There must be an early record in there somewhere to give you all 
> those empty pages, eg if you have one record for 1am 1 January 2016 you 
> will get blanks pages for all in between. 1 January 2016 and 2 December 
> 2017. I will drop you a few queries to run shortly to see exactly what is 
> going on, probably going to have to merge your backup and part of your 
> current data.
>
> Gary
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[gjr80]

Ok, must have been some port filtering on the internet I was using. I suspect 
that the only data in your database is 2nd December onwards only, look at all 
your month and year highs/lows, they are identical. December 2017 must have 
been a memorable month, or you only have data for December 2017. There must be 
an early record in there somewhere to give you all those empty pages, eg if you 
have one record for 1am 1 January 2016 you will get blanks pages for all in 
between. 1 January 2016 and 2 December 2017. I will drop you a few queries to 
run shortly to see exactly what is going on, probably going to have to merge 
your backup and part of your current data.

Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Andrew Milner]

Gary - the link works fine for me so it must be your internet

The issue seems to be that when regenerating the NOAA files they all now 
exist(ie the dropdown is populated), but the data content for all months 
and years up to December 2nd 2017 is missing - the reports are just 
blanks.  Looking at the plots there appears to be data for earlier periods 
in the database (apart from what looks like a glitch around January 2016)

'Tis an interesting one which I'll leave to you - 'cos I don't have a clue 
what could cause this!!  I assume it must be something odd in the DB - 
maybe the summaries need recreating from the archive or something like 
that.  


On Tuesday, 5 December 2017 06:42:40 UTC+2, gjr80 wrote:

> Joe,
>
> I am out at the moment but I am getting unreachable on that link you 
> provided. Could be my internet though.
>
> Gary
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Joe Brockway]

Gary,

What I meant when I said "all the reports regenerated" was the text files 
in the NOAA folder were re-created.

Here is the link to the almanac page on my weewx server. 
http://73.14.239.120:32983/weewx/almanac.html

If you look at the dropdowns for the monthly and yearly summaries (upper 
right corner) there is no data. The problem is now worse that it was this 
morning. 

I have the database backing up every night at midnight to a USB stick on 
the raspberry pi. I did save the database off the USB stick on Nov 26th. 

Hopefully, there is a way to fix the database. Or I could just restore the 
database from the Nov 26th copy.

-Joe
 

On Monday, December 4, 2017 at 7:08:55 PM UTC-7, gjr80 wrote:
>
> Hi,
>
> So when you say *'All the reports regenerated*' do you mean that you now 
> have month and year reports for all months/years you expect and you are 
> just missing all data for 1st December or do you mean something else? We 
> need to be clear on what results you have now as that will determine how to 
> deal with the issue. If you have all reports you expect you should have and 
> are indeed missing output for 1st December then the issue is likely with 
> your database (given that you just forced a complete regenration).
>
> Gary
>
> On Tuesday, 5 December 2017 11:34:20 UTC+10, Joe Brockway wrote:
>>
>> Yes. I am referring to the NOAA reports. 
>>
>> I did stop weewx and delete the NOAA directory in /var/www/weewx. I 
>> started weewx. All the reports regenerated. 
>>
>> However, My yearly report for 2017 only has information for December. 
>> Plus the December 2017 report only has information from December 2 and 
>> forward. I did update weewx to 3.8.0 on December 2nd. My guess is I messed 
>> something up when i updated weewx. Could the issue be in my database? If so 
>> might they be a way to fix the database?
>>
>> -joe
>>
>> On Monday, December 4, 2017 at 5:39:31 AM UTC-7, Tom Keffer wrote:
>>>
>>> I'm not sure what you mean by CLIMATOLOGICAL data. Do you mean the NOAA 
>>> reports? If so, they are generated on a forward going basis: only reports 
>>> more recent than the last one you have will be generated. So, if you're 
>>> missing earlier reports, delete the directory and all will be regenerated.
>>>
>>> -tk
>>>
>>> On Mon, Dec 4, 2017 at 6:33 AM, Joe Brockway  wrote:
>>>
 Well now I have a new issue.

 My CLIMATOLOGICAL data for 2017 only shows December. All the data is 
 there for the other past years. And the data was there before I ran the 
 wee_database command.

 Any suggestions on how to fix this?

 --
 You received this message because you are subscribed to the Google 
 Groups "weewx-user" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to weewx-user+...@googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[gjr80]

Hi,

So when you say *'All the reports regenerated*' do you mean that you now 
have month and year reports for all months/years you expect and you are 
just missing all data for 1st December or do you mean something else? We 
need to be clear on what results you have now as that will determine how to 
deal with the issue. If you have all reports you expect you should have and 
are indeed missing output for 1st December then the issue is likely with 
your database (given that you just forced a complete regenration).

Gary

On Tuesday, 5 December 2017 11:34:20 UTC+10, Joe Brockway wrote:
>
> Yes. I am referring to the NOAA reports. 
>
> I did stop weewx and delete the NOAA directory in /var/www/weewx. I 
> started weewx. All the reports regenerated. 
>
> However, My yearly report for 2017 only has information for December. Plus 
> the December 2017 report only has information from December 2 and forward. 
> I did update weewx to 3.8.0 on December 2nd. My guess is I messed something 
> up when i updated weewx. Could the issue be in my database? If so might 
> they be a way to fix the database?
>
> -joe
>
> On Monday, December 4, 2017 at 5:39:31 AM UTC-7, Tom Keffer wrote:
>>
>> I'm not sure what you mean by CLIMATOLOGICAL data. Do you mean the NOAA 
>> reports? If so, they are generated on a forward going basis: only reports 
>> more recent than the last one you have will be generated. So, if you're 
>> missing earlier reports, delete the directory and all will be regenerated.
>>
>> -tk
>>
>> On Mon, Dec 4, 2017 at 6:33 AM, Joe Brockway  wrote:
>>
>>> Well now I have a new issue.
>>>
>>> My CLIMATOLOGICAL data for 2017 only shows December. All the data is 
>>> there for the other past years. And the data was there before I ran the 
>>> wee_database command.
>>>
>>> Any suggestions on how to fix this?
>>>
>>> --
>>> You received this message because you are subscribed to the Google 
>>> Groups "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to weewx-user+...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Joe Brockway]

Yes. I am referring to the NOAA reports. 

I did stop weewx and delete the NOAA directory in /var/www/weewx. I started 
weewx. All the reports regenerated. 

However, My yearly report for 2017 only has information for December. Plus 
the December 2017 report only has information from December 2 and forward. 
I did update weewx to 3.8.0 on December 2nd. My guess is I messed something 
up when i updated weewx. Could the issue be in my database? If so might 
they be a way to fix the database?

-joe

On Monday, December 4, 2017 at 5:39:31 AM UTC-7, Tom Keffer wrote:
>
> I'm not sure what you mean by CLIMATOLOGICAL data. Do you mean the NOAA 
> reports? If so, they are generated on a forward going basis: only reports 
> more recent than the last one you have will be generated. So, if you're 
> missing earlier reports, delete the directory and all will be regenerated.
>
> -tk
>
> On Mon, Dec 4, 2017 at 6:33 AM, Joe Brockway  > wrote:
>
>> Well now I have a new issue.
>>
>> My CLIMATOLOGICAL data for 2017 only shows December. All the data is 
>> there for the other past years. And the data was there before I ran the 
>> wee_database command.
>>
>> Any suggestions on how to fix this?
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Joe Brockway]

Yes I am referring to the NOAA reports.  I will delete everything in the 
NOAA directory and let it re-create the reports.

Thank you.

On Monday, December 4, 2017 at 5:39:31 AM UTC-7, Tom Keffer wrote:
>
> I'm not sure what you mean by CLIMATOLOGICAL data. Do you mean the NOAA 
> reports? If so, they are generated on a forward going basis: only reports 
> more recent than the last one you have will be generated. So, if you're 
> missing earlier reports, delete the directory and all will be regenerated.
>
> -tk
>
> On Mon, Dec 4, 2017 at 6:33 AM, Joe Brockway  > wrote:
>
>> Well now I have a new issue.
>>
>> My CLIMATOLOGICAL data for 2017 only shows December. All the data is 
>> there for the other past years. And the data was there before I ran the 
>> wee_database command.
>>
>> Any suggestions on how to fix this?
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Joe Brockway]

Yes I am referring to the NOAA reports.  I will delete everything in the 
NOAA directory and let it re-create the reports.

Thanks you.

On Monday, December 4, 2017 at 5:42:28 AM UTC-7, Andrew Milner wrote:
>
> If you are referring to the NOAA reports the simplest way is to delete 
> them all and let weewx recreate them at the next archive interval:
> a) stop weewx
> b) delete all files in NOAA directory
> c) start weewx
>
> as long as the underlying data is in the archive database the NOAA reports 
> will all be recreated as part of the first archive interval processing
>
>
>
> On Monday, 4 December 2017 14:33:28 UTC+2, Joe Brockway wrote:
>
>> Well now I have a new issue.
>>
>> My CLIMATOLOGICAL data for 2017 only shows December. All the data is 
>> there for the other past years. And the data was there before I ran the 
>> wee_database command.
>>
>> Any suggestions on how to fix this? 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Andrew Milner]

If you are referring to the NOAA reports the simplest way is to delete them 
all and let weewx recreate them at the next archive interval:
a) stop weewx
b) delete all files in NOAA directory
c) start weewx

as long as the underlying data is in the archive database the NOAA reports 
will all be recreated as part of the first archive interval processing



On Monday, 4 December 2017 14:33:28 UTC+2, Joe Brockway wrote:

> Well now I have a new issue.
>
> My CLIMATOLOGICAL data for 2017 only shows December. All the data is there 
> for the other past years. And the data was there before I ran the 
> wee_database command.
>
> Any suggestions on how to fix this? 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Thomas Keffer]

I'm not sure what you mean by CLIMATOLOGICAL data. Do you mean the NOAA
reports? If so, they are generated on a forward going basis: only reports
more recent than the last one you have will be generated. So, if you're
missing earlier reports, delete the directory and all will be regenerated.

-tk

On Mon, Dec 4, 2017 at 6:33 AM, Joe Brockway  wrote:

> Well now I have a new issue.
>
> My CLIMATOLOGICAL data for 2017 only shows December. All the data is there
> for the other past years. And the data was there before I ran the
> wee_database command.
>
> Any suggestions on how to fix this?
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Joe Brockway]

On Monday, December 4, 2017 at 5:33:28 AM UTC-7, Joe Brockway wrote:
> Well now I have a new issue.
> 
> My CLIMATOLOGICAL data for 2017 only shows December. All the data is there 
> for the other past years. And the data was there before I ran the 
> wee_database command.
> 
> Any suggestions on how to fix this?

I forgot to menttion when I look at the monthly data for any month in 2017 the 
data is there.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Joe Brockway]

I stopped weewx and re-ran the command:

pi@raspberrypi ~ $ sudo wee_database /etc/weewx/weewx.conf --update

Using configuration file /etc/weewx/weewx.conf

Using database binding 'wx_binding', which is bound to database 
'archive_sqlite'

The update process does not affect archive data, but does alter the 
database.

Continue (y/n)? y

Preparing Interval Weighting Fix...

Daily summary tables are at version 2.0

Interval Weighting Fix is not required.

Preparing Maximum windSpeed Fix...

Updating 'windSpeed' daily summary: 87; Timestamp: 2017-12-03 (1512284400)

Maximum windSpeed Fix completed in 0.75 seconds


Thank you for pointing out that I forgot to stop weewx.



On Sunday, December 3, 2017 at 6:16:23 AM UTC-7, Joe Brockway wrote:
>
> Yes weewx is running. Doh! I will stop weewx and try again. 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Joe Brockway]

Yes weewx is running. Doh! I will stop weewx and try again. 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Updating database error - "weedb.OperationalError: database is locked"

[Thomas Keffer]

Is weewx running?

-tk

On Sat, Dec 2, 2017 at 10:47 AM, Joe Brockway  wrote:

> Hello,
>
>
> I recently updated weewx from 3.6.1 to 3.8.
>
>
> I read the upgrade guide. Updating the database was mentioned. I figured
> might as well make sure the database is good to go. I ran the command
> twice. First time I was told the database was read-only. The second time I
> was told it is locked. What am I missing? I am not getting any other errors
> in the log about the database being locked.
>
>
> Thank you for your help.
>
>
> Joe B
>
>
>
> pi@raspberrypi ~/Downloads $ wee_database /etc/weewx/weewx.conf --update
>
> Using configuration file /etc/weewx/weewx.conf
>
> Using database binding 'wx_binding', which is bound to database
> 'archive_sqlite'
>
> The update process does not affect archive data, but does alter the
> database.
>
> Continue (y/n)? y
>
> Preparing Interval Weighting Fix...
>
> Daily summary tables are at version 1.0
>
> Calculating interval weights...
>
> Interval Weighting Fix completed in 11.00 seconds.
>
> Preparing Maximum windSpeed Fix...
>
> Traceback (most recent call last):
>
>   File "/usr/bin/wee_database", line 793, in 
>
> main()
>
>   File "/usr/bin/wee_database", line 165, in main
>
> update(config_dict, db_binding, options)
>
>   File "/usr/bin/wee_database", line 558, in update
>
> _fix_wind(config_dict, db_binding, options)
>
>   File "/usr/bin/wee_database", line 591, in _fix_wind
>
> wind_obj.run()
>
>   File "/usr/share/weewx/weecfg/database.py", line 196, in run
>
> self.do_fix()
>
>   File "/usr/share/weewx/weecfg/database.py", line 239, in do_fix
>
> day_max, day_max_ts)
>
>   File "/usr/share/weewx/weecfg/database.py", line 331, in write_max
>
> _cursor.execute(max_update_str, (value, when_ts, row_ts))
>
>   File "/usr/share/weewx/weedb/sqlite.py", line 43, in guarded_fn
>
> raise weedb.OperationalError(e)
>
> weedb.OperationalError: attempt to write a readonly database
>
> pi@raspberrypi ~/Downloads $ sudo wee_database /etc/weewx/weewx.conf
> --update
>
> Using configuration file /etc/weewx/weewx.conf
>
> Using database binding 'wx_binding', which is bound to database
> 'archive_sqlite'
>
> The update process does not affect archive data, but does alter the
> database.
>
> Continue (y/n)? y
>
> Preparing Interval Weighting Fix...
>
> Daily summary tables are at version 1.0
>
> Calculating interval weights...
>
> Traceback (most recent call last):e: 2015-07-21 00:25:00 MDT (1437459900)
>
>   File "/usr/bin/wee_database", line 793, in 
>
> main()
>
>   File "/usr/bin/wee_database", line 165, in main
>
> update(config_dict, db_binding, options)
>
>   File "/usr/bin/wee_database", line 552, in update
>
> weight_obj.run()
>
>   File "/usr/share/weewx/weecfg/database.py", line 474, in run
>
> self.dbm.backfill_day_summary()
>
>   File "/usr/share/weewx/weewx/manager.py", line 1477, in
> backfill_day_summary
>
> self._write_metadata('lastUpdate', str(int(lastUpdate)), cursor)
>
>   File "/usr/share/weewx/weewx/manager.py", line 1583, in _write_metadata
>
> (key, value))
>
>   File "/usr/share/weewx/weedb/sqlite.py", line 43, in guarded_fn
>
> raise weedb.OperationalError(e)
>
> weedb.OperationalError: database is locked
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.