[weewx-user] Re: table archive_day_barometer already exists

[Dave Harper]

Phil (and anyone else interested),

The script is now written and I have spent most of the day testing it.  It 
has currently been running for about three hours now and seems to be 
running correctly.  I'm attaching the script and a set of instructions on 
how to configure and run it.  The script creates and maintains a "last 
known good" copy of the weewx.sdb database at whatever interval the user 
specifies.  It also attempts to be as robust as possible, notifying the 
user (via email) of any exception conditions rather than simply "failing 
silently".  If you (or anyone else) runs it, I'd be interested in getting 
feedback as to any issues discovered.

Dave

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Instructions.pdf
Description: Adobe PDF document
#! /usr/bin/env python3


#  #
#  sdbCheck.py #
#  #
# The primary purpose of this script is to create and maintain a "last known   #
# good" copy of the weeWX database.  To do this, it makes a copy of the weeWX  #
# database in the script starting directory (waiting, if necessary, until the  #
# database is not being updated).  The integrity of the copy is checked using  #
# sqlite3 and, if it is good, it replaces the previous copy in the ./Backup#
# directory.  If a problem is found an email is sent to the user.  A small log #
# file is also created indicating the results of the run.  #
#  #
# The script is intended to be run periodically by the operating system using  #
# whatever support the OS provides (cron, Task Scheduler, etc.).  It is also   #
# intended to run as "safely" as possible without user intervention, so the#
# first thing the script does is to verify that the previous run happened when #
# it was supposed to and that it completed correctly.  If not, an email is #
# sent to the user.# 
#  #
# There are a number of script options that can be configured by the user, #
# such as email address, period of weeWX updates, etc.  These are listed below #
# and must be set prior to operation.  To assist in verifying that they are#
# set correctly, the script can be manually started on the command line with   #
# anything as a parameter.  When the script is started, it checks the length   #
# of sys.argv and, if it is more than 1, it will perform a configuration check #
# and then exit after printing the results on the screen and sending a test#
# email.   #
#  #
# For support issues, please contact d...@dlharper.net.  Feel free to modify   #
# and / or redistribute this script.  If you add a useful feature, please let  #
# me know. #
#  #




# Import the modules that will be used throughout this script. #


import sys
import os
import os.path
import time
from datetime import datetime
import shutil
from shutil import copyfile



# The following options can be used to tailor the script to a specific #
# environment. #


WEEWX_DIR = ""  # location of the weewx.sdb file
WEEWX_UPDATE = -1  # number of seconds between weewx.sdb updates
SCRIPT_UPDATE = -1   # number of seconds between runs of this script
INSTALL_DIR = ""   # full path to where this script is installed
EMAIL_ADDR = ""# login address on email host
EMAIL_PASSWD = ""# email host login password
EMAIL_HOST = "" # email host
EMAIL_PORT = -1

[weewx-user] Re: table archive_day_barometer already exists

[Dave Harper]

Phil,

No problem.  Just starting it now and as soon as it's done and tested, I'l 
post a copy.

Dave

On Wednesday, December 12, 2018 at 3:43:46 PM UTC-6, kutz...@gmail.com 
wrote:
>
> I'd appreciate you sharing a copy of the script with me after you write 
> it. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: table archive_day_barometer already exists

[Dave Harper]

Phil,

One of the things I will be implementing in my new backup strategy is a 
check to make sure the database isn't corrupted.  This can be done using 
command line parameters to sqlite3.  When run manually, it looks something 
like:
pi@Weather:~ $ sqlite3 weewx.sdb 'pragma integrity_check'
ok
pi@Weather:~ $
I will be including this in a new Python script which will send me an email 
if it detects corruption.  Otherwise, it will archive that database as 
"known good".  This should give me an early warning if something goes 
wrong.  

Dave
  

On Wednesday, December 12, 2018 at 10:36:09 AM UTC-6, kutz...@gmail.com 
wrote:
>
> In the meantime, I can increase the time between rsync copies, as my Davis 
> console should buffer about  a week's worth of archive data. That would 
> give me more time to react to any database corruption.
>
> phil
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: table archive_day_barometer already exists

[Dave Harper]

Gary,

The backup strategy I use is a Python script I wrote that is started by 
cron every night.  I had read that there could be a problem if the database 
is copied while it's being updated.  The database is updated once a minute 
by weeWX so the script monitors the mtime of the weewx.sdb file, waits for 
the timestamp to change then waits about 15 seconds more before copying it 
to a subdirectory off my home directory.  I also copy the /etc/weewx 
directory into the same backup subdirectory and then the script calls 
Duplicity to do the actual backup.  Duplicity works by starting with a full 
backup the first time around and then doing incrementals from then on.  A 
restore starts with the original full backup and then applies all 
incremental changes to get to the target restore date.  To keep a restore 
process within reason, I only do a month of incrementals and on the first 
of each month the script creates a new monthly directory on the NAS.  This 
strategy has seemed to work well over the past year or so and looking into 
why it failed this time around will be one of the top priorities for 
tomorrow.

Dave

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: table archive_day_barometer already exists

[Dave Harper]

Gary - I am in your debt.  The --reconfigure option seems to have done the 
trick.  It took a while to generate the new file but, once it was done, 
weeWX was able to accept the database.  It then spend another large chunk 
of time rebuilding the daily summaries and, once that was done, it started 
downloading the updates from my console.  It has loaded almost 7 hours 
worth and has about 3 more to go but everything seems to be going smoothly 
and there is no reason to think it won't finish normally at this point.  
I've been running on a dedicated Raspberry Pi and a while back, in 
anticipation of this crash, I built another Pi that boots from a 250GB hard 
drive and installed everything on that.  Now that I have a working database 
again, I will be switching over to that one tomorrow and implementing 
another level of backups in addition to the NAS I have now.  I really 
appreciate the time you've spent helping me get up and going again.

Regards,
Dave

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: table archive_day_barometer already exists

[Dave Harper]

Thanks to you both for your suggestions.  Gary, I followed your 
instructions with the following results:
1) Stopped weeWX and verified in /var/log/syslog that it was stopped
2) Used the wee_database utility to drop the daily summaries.  The system 
responded with:
Dropping daily summary tables from 'weewx.sdb' ...  
No daily summaries found in database 'weewx.sdb'. Nothing done.
3) Tried the --rebuild-daily.  That produced the following list of errors:
Traceback (most recent call last):
  File "/usr/bin/wee_database", line 793, in 
main()
  File "/usr/bin/wee_database", line 153, in main
rebuildDaily(config_dict, db_binding, options)
  File "/usr/bin/wee_database", line 251, in rebuildDaily
with weewx.manager.open_manager_with_config(config_dict, db_binding, 
initialize=True) as dbmanager:
  File "/usr/share/weewx/weewx/manager.py", line 1017, in 
open_manager_with_config
return open_manager(manager_dict, initialize)
  File "/usr/share/weewx/weewx/manager.py", line 1006, in open_manager
manager_dict['schema'])
  File "/usr/share/weewx/weewx/manager.py", line 140, in open_with_create
dbmanager = cls(connection, table_name=table_name, schema=schema)
  File "/usr/share/weewx/weewx/manager.py", line 1152, in __init__
self._initialize_day_tables(schema, _cursor)
  File "/usr/share/weewx/weewx/wxmanager.py", line 35, in 
_initialize_day_tables
weewx.manager.DaySummaryManager._initialize_day_tables(self, 
archiveSchema, cursor)
  File "/usr/share/weewx/weewx/manager.py", line 1179, in 
_initialize_day_tables
cursor.execute(DaySummaryManager.sql_create_str % (self.table_name, 
_obs_type))
  File "/usr/share/weewx/weedb/sqlite.py", line 39, in guarded_fn
raise weedb.TableExistsError(e)
weedb.TableExistsError: table archive_day_barometer already exists

I also played with some of the other options to see what they would do 
(using the --dry-run option just in case) but they all ended up printing a 
list of errors that terminated in:
weedb.OperationalError: No day summary schema for table 'archive' in 
database 'weewx.sdb'

Still digging...


-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] table archive_day_barometer already exists

[Dave Harper]

Over the past year I've had numerous cases of a corrupted database.  I 
learned early on to have a good backup scheme in place and it has served me 
well - up until today.  After I noticed that the web page was no longer 
updating I checked the /var/sys/logfile and found the message 
"DatabaseError: database disk image is malformed".  Shortly after that I 
discovered my backups for the past month were not working, so I've been 
faced with trying to repair the weewx.sdb file.  I've found several posts 
describing how to recover from problems like this and ended up using the 
repairsdb.sh script that I found in one of them.  That fixed the 
"malformed" error and I am now able to load the database but when weewx 
starts it now aborts with the error that "table archive_day_barometer 
already exists".  I had never used sqlite3 prior to weewx and am not really 
familiar with what to try now.  Anybody have any ideas I can try?

Thanks
Dave

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] weewx not starting after reboot (manual intervention necessary)

[Dave Harper]

On Friday, September 22, 2017 at 11:42:32 PM UTC-5, Andrew Milner wrote:
>
> I think it has started when wifi network was up rather than Ethernet 
> network.  Did you set wait for network in raspi-config - which is the RPi 
> method of ensuring network is available while booting.
>
>>
>> Thanks for the suggestion - I was unaware of this option.  However, I 
tried it (independently from the suggestion Tom made) and for some reason 
it didn't work (which surprised me as it sounded like an obvious solution). 
 Looking at the syslog after the boot showed almost exactly the same 
situation as shown in the syslog1 file I attached in the original post - 
weewx still started up before networking was up.  After the boot I went 
back in to raspi-config and verified that it was set to wait and it was. 
 As for wifi, that is turned off.  The Pi is physically located next to the 
Vantage Pro2 console which already required wired Ethernet for uploading so 
I went wired for the Pi too.

Dave 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] weewx not starting after reboot (manual intervention necessary)

[Dave Harper]

On Friday, September 22, 2017 at 9:25:58 PM UTC-5, Tom Keffer wrote:
>
> Sorry. The WLIP logger is a common nickname for the Davis "WeatherLink IP 
> logger".  Like this 
> .
>
>
I'm officially embarrassed - apparently I was more tired last night than I 
realized. Before calling it a night I decided to Google "Vantage logger" 
and realized my mistake about the time your followup post arrived (and yes, 
that's what I have).  My first thought had been "Hmmm... must be some type 
of Linux logging tool I'm not familiar with".  Anyway, this morning I made 
the modification you suggested and it worked perfectly.  Upon reboot, the 
connection still could not be opened, however this time the error message 
was replaced with one that said it was waiting 60 seconds and would try 
again.  The second time around, it connected with no problems.  Many thanks 
for your help on resolving this.

Dave

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] weewx not starting after reboot (manual intervention necessary)

[Dave Harper]

On Friday, September 22, 2017 at 9:11:19 PM UTC-5, Tom Keffer wrote:
>
>
> One question: I assume you have the WLIP version of the Vantage logger?
>
>
I'm just starting to transition from Windows to Linux and still have a lot 
to learn.  For example, I don't know what the Vantage logger is or what the 
WLIP version might be.  This is good exercise though - I'll dig into it and 
my next reply will try to contain some useful information regarding it. 
 However, it's starting to get late where I am and I probably need to call 
it a day before I break something.  I'll get after it in the morning and 
should have something to report back then.  Thanks for the pointers.

Dave

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] weewx not starting after reboot (manual intervention necessary)

[Dave Harper]

A couple of days ago I switched from a Windows 7 system running Weatherlink 
to a Raspberry Pi 3B running weewx under the jessie distribution.  Once 
running, the program works great but there is one problem I must solve and 
I'm not sure how to do it.  If the Pi reboots weewx does not start 
correctly.  Looking at /var/log/syslog (attached syslog1) I found that 
weewx was starting before networking was up and going.  Thus, it got an 
error trying to access the console and exited (right after the exit, 
networking starts up).  I searched to see if there was a way to delay weewx 
from starting until networking was up and found a post in another forum 
that indicated I could delay the start of weewx by adding "$network" to the 
"# Required-Start" line in init.d/weewx.  I did that and the result was the 
boot shown in syslog2.  It appears that weewx now waits until networking is 
up, but the problem remains.  Once the system is up and running I can stop 
weewx, then start it again and, "tailing" the /var/log/syslog file I can 
see that this time it starts up okay.  Unfortunately, I might not be around 
when a power hit occurs, so I'd like to find out what's going wrong. 
 Anyone have any ideas I can try?

Thanks,
Dave

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


syslog1
Description: Binary data


syslog2
Description: Binary data