See the wiki article Cleaning up old "bad" data <https://github.com/weewx/weewx/wiki/Cleaning-up-old-'bad'-data>, in particular, the section *Deleting unwanted records <https://github.com/weewx/weewx/wiki/Cleaning-up-old-'bad'-data#deleting-unwanted-records>* .
On Sun, Jul 24, 2022 at 1:53 AM Fische Namenlos <aalmeida2...@gmail.com> wrote: > great, > it worked. my linux-computer was working for hours for only 60MB of data, > but database is readable by weewx. > > a single gap is now that the NOAA-files are created from up 1988 and, of > course, empty. The station has started in 2019. > Is there a way to fix it? > > regards > > Oscar schrieb am Dienstag, 12. Juli 2022 um 17:47:09 UTC+2: > >> The database is corrupt. Could be caused by a bad SD card, losing power >> during a write, a number of reasons. I've been able to use this process to >> recover most of a corrupt data base. With 5 years of data, the recreate >> process will take a long time (days(?)) on a pi. I recommend coping the >> database to a faster machine to create the database, then copy it back to >> the pi. >> >> # Dump the database as INSERT commands. >> sqlite3 weewx.sdb >> .mode insert >> .output dump_all.sql >> .dump >> .exit >> >> # remove unwanted records from the dump file >> >> cat dump_all.sql | grep -v TRANSACTION | grep -v ROLLBACK | grep -v >> COMMIT >dump_all_notrans.sql >> >> # recreate the database using the insert statements. >> >> sqlite3 new-weewx.sdb ".read dump_all_notrans.sql" >> >> make a copy of your existing weewx.sdb file and then copy new-weewx.sdb >> to weewx.sdb. >> >> >> On Monday, July 11, 2022 at 7:02:52 AM UTC-6 tke...@gmail.com wrote: >> >>> It is very unlikely that the size of the database is a problem. Like >>> Graham, I have over 15 years of data on mine without a problem. >>> >>> You did not say what kind of storage you are using, but if it is an old >>> SD card, it is much more likely that the card is corrupt. >>> >>> You may be able to recover the database by using the utility sqlite3. >>> >>> sqlite3 /home/weewx/archive/weewx.sdb >>> sqlite> .recover >>> sqlite> .quit >>> >>> However, usually this does not work, in which case you will have to >>> recover from a backup copy. Then, replace the card. >>> >>> -tk >>> >>> >>> On Sun, Jul 10, 2022 at 10:27 AM Fische Namenlos <aalmei...@gmail.com> >>> wrote: >>> >>>> here it is: >>>> Sun Jul 10 19:25:39 2022 user.debug weewx[2812] DEBUG weewx.restx: Shut >>>> down StationRegistry thread. >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> Caught unrecoverable exception: >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** database disk image is malformed >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** Traceback (most recent call last): >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewx/engine.py", line 214, in run >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** self.dispatchEvent(weewx.Event(weewx.CHECK_LOOP, packet=packet)) >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewx/engine.py", line 245, in dispatchEvent >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** callback(event) >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewx/engine.py", line 634, in check_loop >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** raise BreakLoop >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** weewx.engine.BreakLoop >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** During handling of the above exception, another exception occurred: >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** Traceback (most recent call last): >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewxd", line 153, in main >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** engine.run() >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewx/engine.py", line 221, in run >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** self.dispatchEvent(weewx.Event(weewx.POST_LOOP)) >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewx/engine.py", line 245, in dispatchEvent >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** callback(event) >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewx/engine.py", line 644, in post_loop >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** self._software_catchup() >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewx/engine.py", line 715, in >>>> _software_catchup >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** self.engine.dispatchEvent(weewx.Event(weewx.NEW_ARCHIVE_RECORD, >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewx/engine.py", line 245, in dispatchEvent >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** callback(event) >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewx/engine.py", line 675, in >>>> new_archive_record >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** dbmanager.addRecord(event.record, >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewx/manager.py", line 301, in addRecord >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** self._updateHiLo(accumulator, cursor) >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewx/manager.py", line 1031, in _updateHiLo >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** _stats_dict = self._get_day_summary(_sod_ts, cursor) >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weewx/manager.py", line 1412, in >>>> _get_day_summary >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** _cursor.execute( >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weedb/sqlite.py", line 30, in guarded_fn >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** return fn(*args, **kwargs) >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** File "/home/weewx/bin/weedb/sqlite.py", line 219, in execute >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** return sqlite3.Cursor.execute(self, *args, **kwargs) >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** sqlite3.DatabaseError: database disk image is malformed >>>> Sun Jul 10 19:25:39 2022 user.crit weewx[2812] CRITICAL __main__: >>>> **** Exiting. >>>> peterq...@gmail.com schrieb am Sonntag, 10. Juli 2022 um 17:45:42 >>>> UTC+2: >>>> >>>>> What error messages are you getting? >>>>> >>>>> On Sun, Jul 10, 2022 at 8:00 AM Graham Eddy <g...@geddy.au> wrote: >>>>> >>>>>> 15 years of data, running happily on RPi 4B: >>>>>> >>>>>> *graham@paperbark*:*/home/weewx/archive $* ls -l weewx.sdb >>>>>> -rw-r--r-- 1 weewx weewx 175935488 Jul 11 00:55 weewx.sdb >>>>>> >>>>>> >>>>>> On 11 Jul 2022, at 12:43 am, Fische Namenlos <aalmei...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> my station is using sqlite database (weewx.sdb) for nearly 5 years, >>>>>> and now - I suppose - is getting to big for my Raspi-server, it stops >>>>>> working/writting. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to weewx-user+...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-user/8C3EB25A-F4A3-4D14-B43C-16556831AF91%40geddy.au >>>>>> <https://groups.google.com/d/msgid/weewx-user/8C3EB25A-F4A3-4D14-B43C-16556831AF91%40geddy.au?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> >>>>> >>>>> -- >>>>> Peter Quinn >>>>> (415)794-2264 <(415)%20794-2264> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to weewx-user+...@googlegroups.com. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/47d68a21-a20c-4145-bfc0-2179b9e847e8n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/weewx-user/47d68a21-a20c-4145-bfc0-2179b9e847e8n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/ed4161fb-8421-4f40-abcf-0f2323425010n%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-user/ed4161fb-8421-4f40-abcf-0f2323425010n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zECZjWXrpXpLVPw-XVMJ2%2BytYhZ83-n2HNQza3VbsRk2Lg%40mail.gmail.com.
