[weewx-user] Re: wee_import from WD logs fails
Hi Gary, I'm going to start it all over again this weekend, when having time. I organized all logdata from each year into a separate directory. When importing it, I just edit the import path in the configfile and then start wee_import with --import-config=myimportconfig.con --verbose, and the let it run Here is some 6 years of data, some newer and some older. You can play around with it, if you like. https://drive.google.com/drive/folders/1zwvlJSlGc0gtypuBaFIYq0V-jCg1e776?usp=sharing I can just upload the whole bunch of it, if you want. Thank you for your help!!! Peter gjr80 schrieb am Montag, 15. Februar 2021 um 21:48:22 UTC+1: > Hi Peter, > > Before you start it off again can you describe how you are doing the > import; ie time period per import, one big import, multiple smaller > periods? How is you data organised on disc, one large directory with all > file, year by year directories etc. > > Gary > > On Tuesday, 16 February 2021 at 00:00:27 UTC+10 peter.su...@gmail.com > wrote: > >> Hi Gary, >> >> today my Raspberry 4 arrived, which will hopefully be my long-running >> weather computer. >> I tried to import all my data on Pi3. It worked back for about 8 years, >> but then the performance crashed into hell. >> The import of one year took about 1 week and was getting worse. >> >> Now I'm starting it all over on a fresh install and report, what will be >> the result. >> >> Peter >> gjr80 schrieb am Montag, 1. Februar 2021 um 23:28:04 UTC+1: >> >>> Hi Peter, >>> >>> On Tuesday, 2 February 2021 at 06:03:36 UTC+10 peter.su...@gmail.com >>> wrote: >>> Hey Gary, its looking perfect. I just separated the logs for each year and right now I am testing the 6th year in a row, doing a quick dry-run. The script is finding some duplicates, but it looks like every complete year from 01/01 to 12/31 should be imported into the database. >>> >>> That is interesting, the screen output you posted originally clearly >>> showed an error decoding a date-time, I am surprised that re-arranging the >>> log files fixed that. The first person that trialled the WD import module >>> was in a similar boat to you; they had many years of WD data and when they >>> first tried to import the data they did it in one run and not only did it >>> take many hours but wee_import eventually ground to a halt and crashed >>> (the exact details escape me). So importing/processing a year at a time may >>> be more manageable and safer. >>> >>> The duplicates issue is not uncommon, the few suites of WD logs I have >>> worked on all contained the odd duplicate date-time. wee_import will >>> simply ignore the duplicates, the only issue is when the data for each >>> duplicate is different. Probably not going to make too much difference over >>> 15 years of data at one minute intervals. >>> >>> One more question from my side: Would you prefer a MySQL database with all this mass of data? >>> >>> I think the consensus is that unless you have good reason to use >>> MySQL/MariaDB stick with SQLite. SQLite is faster and simpler to >>> backup/restore. SQLite will easy handle all of your 15 years of data. >>> >>> Gary >>> >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6c84f279-cf8b-40df-a84f-7bb6032867bfn%40googlegroups.com.
[weewx-user] Re: wee_import from WD logs fails
Hi Peter, Before you start it off again can you describe how you are doing the import; ie time period per import, one big import, multiple smaller periods? How is you data organised on disc, one large directory with all file, year by year directories etc. Gary On Tuesday, 16 February 2021 at 00:00:27 UTC+10 peter.su...@gmail.com wrote: > Hi Gary, > > today my Raspberry 4 arrived, which will hopefully be my long-running > weather computer. > I tried to import all my data on Pi3. It worked back for about 8 years, > but then the performance crashed into hell. > The import of one year took about 1 week and was getting worse. > > Now I'm starting it all over on a fresh install and report, what will be > the result. > > Peter > gjr80 schrieb am Montag, 1. Februar 2021 um 23:28:04 UTC+1: > >> Hi Peter, >> >> On Tuesday, 2 February 2021 at 06:03:36 UTC+10 peter.su...@gmail.com >> wrote: >> >>> Hey Gary, >>> its looking perfect. I just separated the logs for each year and right >>> now I am testing the 6th year in a row, doing a quick dry-run. >>> The script is finding some duplicates, but it looks like every complete >>> year from 01/01 to 12/31 should be imported into the database. >>> >> >> That is interesting, the screen output you posted originally clearly >> showed an error decoding a date-time, I am surprised that re-arranging the >> log files fixed that. The first person that trialled the WD import module >> was in a similar boat to you; they had many years of WD data and when they >> first tried to import the data they did it in one run and not only did it >> take many hours but wee_import eventually ground to a halt and crashed >> (the exact details escape me). So importing/processing a year at a time may >> be more manageable and safer. >> >> The duplicates issue is not uncommon, the few suites of WD logs I have >> worked on all contained the odd duplicate date-time. wee_import will >> simply ignore the duplicates, the only issue is when the data for each >> duplicate is different. Probably not going to make too much difference over >> 15 years of data at one minute intervals. >> >> One more question from my side: Would you prefer a MySQL database with >>> all this mass of data? >>> >> >> I think the consensus is that unless you have good reason to use >> MySQL/MariaDB stick with SQLite. SQLite is faster and simpler to >> backup/restore. SQLite will easy handle all of your 15 years of data. >> >> Gary >> > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c9b2b67d-9377-488f-8d5c-94662cf13315n%40googlegroups.com.
[weewx-user] Re: wee_import from WD logs fails
Hi Gary, today my Raspberry 4 arrived, which will hopefully be my long-running weather computer. I tried to import all my data on Pi3. It worked back for about 8 years, but then the performance crashed into hell. The import of one year took about 1 week and was getting worse. Now I'm starting it all over on a fresh install and report, what will be the result. Peter gjr80 schrieb am Montag, 1. Februar 2021 um 23:28:04 UTC+1: > Hi Peter, > > On Tuesday, 2 February 2021 at 06:03:36 UTC+10 peter.su...@gmail.com > wrote: > >> Hey Gary, >> its looking perfect. I just separated the logs for each year and right >> now I am testing the 6th year in a row, doing a quick dry-run. >> The script is finding some duplicates, but it looks like every complete >> year from 01/01 to 12/31 should be imported into the database. >> > > That is interesting, the screen output you posted originally clearly > showed an error decoding a date-time, I am surprised that re-arranging the > log files fixed that. The first person that trialled the WD import module > was in a similar boat to you; they had many years of WD data and when they > first tried to import the data they did it in one run and not only did it > take many hours but wee_import eventually ground to a halt and crashed > (the exact details escape me). So importing/processing a year at a time may > be more manageable and safer. > > The duplicates issue is not uncommon, the few suites of WD logs I have > worked on all contained the odd duplicate date-time. wee_import will > simply ignore the duplicates, the only issue is when the data for each > duplicate is different. Probably not going to make too much difference over > 15 years of data at one minute intervals. > > One more question from my side: Would you prefer a MySQL database with all >> this mass of data? >> > > I think the consensus is that unless you have good reason to use > MySQL/MariaDB stick with SQLite. SQLite is faster and simpler to > backup/restore. SQLite will easy handle all of your 15 years of data. > > Gary > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d6822694-7e91-42a2-b2d4-b661bf376b49n%40googlegroups.com.
[weewx-user] Re: wee_import from WD logs fails
Hi Peter, On Tuesday, 2 February 2021 at 06:03:36 UTC+10 peter.su...@gmail.com wrote: > Hey Gary, > its looking perfect. I just separated the logs for each year and right now > I am testing the 6th year in a row, doing a quick dry-run. > The script is finding some duplicates, but it looks like every complete > year from 01/01 to 12/31 should be imported into the database. > That is interesting, the screen output you posted originally clearly showed an error decoding a date-time, I am surprised that re-arranging the log files fixed that. The first person that trialled the WD import module was in a similar boat to you; they had many years of WD data and when they first tried to import the data they did it in one run and not only did it take many hours but wee_import eventually ground to a halt and crashed (the exact details escape me). So importing/processing a year at a time may be more manageable and safer. The duplicates issue is not uncommon, the few suites of WD logs I have worked on all contained the odd duplicate date-time. wee_import will simply ignore the duplicates, the only issue is when the data for each duplicate is different. Probably not going to make too much difference over 15 years of data at one minute intervals. One more question from my side: Would you prefer a MySQL database with all > this mass of data? > I think the consensus is that unless you have good reason to use MySQL/MariaDB stick with SQLite. SQLite is faster and simpler to backup/restore. SQLite will easy handle all of your 15 years of data. Gary -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b4d2b792-3cbb-4fb5-9964-d3b5fbdcca0en%40googlegroups.com.
[weewx-user] Re: wee_import from WD logs fails
Hey Gary, its looking perfect. I just separated the logs for each year and right now I am testing the 6th year in a row, doing a quick dry-run. The script is finding some duplicates, but it looks like every complete year from 01/01 to 12/31 should be imported into the database. One more question from my side: Would you prefer a MySQL database with all this mass of data? Thank you very much! Peter Peter Süß schrieb am Montag, 1. Februar 2021 um 06:42:53 UTC+1: > Thank you Gary. > This might be the error from my side. I just copied the whole bunch of > logs into my import directory. > I will make subdir for every year, then I am going to test again. > Maybe going backward from 2020 to 2005 might be a good idea to find out my > problems. > > Version of my WeeWX is 4.4.0, running on a fresh Raspian image, installed > on Friday. > > Let me test tonight and I will report you back. > > Peter > > gjr80 schrieb am Montag, 1. Februar 2021 um 04:38:54 UTC+1: > >> Peter, >> >> I ran wee_import under WeeWX v4.4.0/python3 using the import config file >> and log files posted and the import processed without error: >> >> gary@buster31:~$ /home/weewx/bin/wee_import >> --import-config=/var/tmp/wd-import-test.conf --from=2020-12-01T00:00 >> --to=2020-12-31T23:59 --verbose >> Using WeeWX configuration file /home/weewx/weewx.conf >> >> Starting wee_import... >> Weather Display monthly log files in the '/tmp/wdisplay' directory will >> be imported >> The following options will be used: >> config=/home/weewx/weewx.conf, >> import-config=/var/tmp/wd-import-test.conf >> >> from=2020-12-01T00:00, to=2020-12-31T23:59 >> dry-run=None, calc_missing=True, ignore_invalid_data=True >> monthly logs are in METRIC units >> tranche=250, interval=1 >> UV=True, radiation=True ignore extreme temperature and humidity=True >> Using database binding 'wx_binding', which is bound to database >> 'weewx.sdb' >> Destination table 'archive' unit system is '0x01' (US). >> Missing derived observations will be calculated. >> Observations timestamped after 2020-12-01 00:00:00 AEST (1606744800) and >> up to and >> including 2020-12-31 23:59:00 AEST (1609423140) will be imported. >> >> Starting import ... >> Records covering multiple periods have been identified for import. >> Obtaining raw import data for period 1 ... >> Raw import data read successfully for period 1. >> Mapping raw import data for period 1 ... >> Mapped 44639 records. >> Raw import data mapped successfully for period 1. >> Saving mapped data to archive for period 1 ... >> 44639 records identified for import. >> Proceeding will save all imported records in the WeeWX archive. >> Are you sure you want to proceed (y/n)? y >> Unique records processed: 44639; Last timestamp: 2020-12-31 23:59:00 AEST >> (1609423140) >> Mapped data saved to archive successfully for period 1. >> Calculating missing derived observations ... >> Processing record: 44639; Last record: 2021-01-01 00:00:00 AEST >> (1609423200) >> Recalculating daily summaries... >> Records processed: 44000; time: 2020-12-31 13:20:00 AEST (1609384800) >> Finished recalculating daily summaries >> Finished calculating missing derived observations >> Finished import >> 44639 records were processed and 44639 unique records imported in 249.23 >> seconds. >> Those records with a timestamp already in the archive will not have been >> imported. Confirm successful import in the WeeWX log file. >> >> So I am confident there is nothing wrong with the import config file and >> log files posted. Granted it was a setup.py install but should not >> really make a difference to wee_import. >> >> You said you had 15 years of WD data, when you did the import were there >> other older log files in /tmp/wdisplay? The reason I ask is that whilst >> you can limit the time frame over which wee_import imports data, all >> that does is apply a filter to the final processing/saving of records. >> wee_import will still go through and read all log files it finds >> starting at the oldest. So if you had 15 years of WD data in a directory >> and tried to import the most recent month all 15 years of log files will be >> read and processed in a limited manner. The initial processing of the raw >> import data involves assembling the individual date and time components >> from each row in the log. That appears to be where your import failed. When >> I wrote the WD component of wee_import I found a number of data >> inconsistencies in the WD log files I was using so I guess it is possible >> you may have similar inconsistencies. Have a look through the oldest WD log >> file set in /tmp/wdisplay and see if you can spot an obvious >> error/inconsistency in the date and time data. If you can't spot anything >> could you post the respective log files here. Alternatively, if you let me >> know what version of WeeWX you are running I can provide you with an >> instrumented version of wee_import to log details o
[weewx-user] Re: wee_import from WD logs fails
Thank you Gary. This might be the error from my side. I just copied the whole bunch of logs into my import directory. I will make subdir for every year, then I am going to test again. Maybe going backward from 2020 to 2005 might be a good idea to find out my problems. Version of my WeeWX is 4.4.0, running on a fresh Raspian image, installed on Friday. Let me test tonight and I will report you back. Peter gjr80 schrieb am Montag, 1. Februar 2021 um 04:38:54 UTC+1: > Peter, > > I ran wee_import under WeeWX v4.4.0/python3 using the import config file > and log files posted and the import processed without error: > > gary@buster31:~$ /home/weewx/bin/wee_import > --import-config=/var/tmp/wd-import-test.conf --from=2020-12-01T00:00 > --to=2020-12-31T23:59 --verbose > Using WeeWX configuration file /home/weewx/weewx.conf > > Starting wee_import... > Weather Display monthly log files in the '/tmp/wdisplay' directory will be > imported > The following options will be used: > config=/home/weewx/weewx.conf, > import-config=/var/tmp/wd-import-test.conf > > from=2020-12-01T00:00, to=2020-12-31T23:59 > dry-run=None, calc_missing=True, ignore_invalid_data=True > monthly logs are in METRIC units > tranche=250, interval=1 > UV=True, radiation=True ignore extreme temperature and humidity=True > Using database binding 'wx_binding', which is bound to database 'weewx.sdb' > Destination table 'archive' unit system is '0x01' (US). > Missing derived observations will be calculated. > Observations timestamped after 2020-12-01 00:00:00 AEST (1606744800) and > up to and > including 2020-12-31 23:59:00 AEST (1609423140) will be imported. > > Starting import ... > Records covering multiple periods have been identified for import. > Obtaining raw import data for period 1 ... > Raw import data read successfully for period 1. > Mapping raw import data for period 1 ... > Mapped 44639 records. > Raw import data mapped successfully for period 1. > Saving mapped data to archive for period 1 ... > 44639 records identified for import. > Proceeding will save all imported records in the WeeWX archive. > Are you sure you want to proceed (y/n)? y > Unique records processed: 44639; Last timestamp: 2020-12-31 23:59:00 AEST > (1609423140) > Mapped data saved to archive successfully for period 1. > Calculating missing derived observations ... > Processing record: 44639; Last record: 2021-01-01 00:00:00 AEST > (1609423200) > Recalculating daily summaries... > Records processed: 44000; time: 2020-12-31 13:20:00 AEST (1609384800) > Finished recalculating daily summaries > Finished calculating missing derived observations > Finished import > 44639 records were processed and 44639 unique records imported in 249.23 > seconds. > Those records with a timestamp already in the archive will not have been > imported. Confirm successful import in the WeeWX log file. > > So I am confident there is nothing wrong with the import config file and > log files posted. Granted it was a setup.py install but should not really > make a difference to wee_import. > > You said you had 15 years of WD data, when you did the import were there > other older log files in /tmp/wdisplay? The reason I ask is that whilst > you can limit the time frame over which wee_import imports data, all that > does is apply a filter to the final processing/saving of records. > wee_import will still go through and read all log files it finds starting > at the oldest. So if you had 15 years of WD data in a directory and tried > to import the most recent month all 15 years of log files will be read and > processed in a limited manner. The initial processing of the raw import > data involves assembling the individual date and time components from each > row in the log. That appears to be where your import failed. When I wrote > the WD component of wee_import I found a number of data inconsistencies > in the WD log files I was using so I guess it is possible you may have > similar inconsistencies. Have a look through the oldest WD log file set in > /tmp/wdisplay and see if you can spot an obvious error/inconsistency in > the date and time data. If you can't spot anything could you post the > respective log files here. Alternatively, if you let me know what version > of WeeWX you are running I can provide you with an instrumented version of > wee_import to log details of the offending record/log. > > Gary > On Monday, 1 February 2021 at 00:59:35 UTC+10 peter.su...@gmail.com wrote: > >> Hi Gary, >> >> thanks for your quick answer. >> Please check the attached file. All I edited was the path to data and >> filenames. I limited it to only one type of file, with my last change. >> >> No, problem when it takes time. >> Thanks for checking out >> >> Peter >> gjr80 schrieb am Sonntag, 31. Januar 2021 um 13:58:21 UTC+1: >> >>> Hi, >>> >>> Nothing leaps out at me having a quick look at the output and log files >>> you posted. I'll endeavour to ha
[weewx-user] Re: wee_import from WD logs fails
Peter, I ran wee_import under WeeWX v4.4.0/python3 using the import config file and log files posted and the import processed without error: gary@buster31:~$ /home/weewx/bin/wee_import --import-config=/var/tmp/wd-import-test.conf --from=2020-12-01T00:00 --to=2020-12-31T23:59 --verbose Using WeeWX configuration file /home/weewx/weewx.conf Starting wee_import... Weather Display monthly log files in the '/tmp/wdisplay' directory will be imported The following options will be used: config=/home/weewx/weewx.conf, import-config=/var/tmp/wd-import-test.conf from=2020-12-01T00:00, to=2020-12-31T23:59 dry-run=None, calc_missing=True, ignore_invalid_data=True monthly logs are in METRIC units tranche=250, interval=1 UV=True, radiation=True ignore extreme temperature and humidity=True Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Destination table 'archive' unit system is '0x01' (US). Missing derived observations will be calculated. Observations timestamped after 2020-12-01 00:00:00 AEST (1606744800) and up to and including 2020-12-31 23:59:00 AEST (1609423140) will be imported. Starting import ... Records covering multiple periods have been identified for import. Obtaining raw import data for period 1 ... Raw import data read successfully for period 1. Mapping raw import data for period 1 ... Mapped 44639 records. Raw import data mapped successfully for period 1. Saving mapped data to archive for period 1 ... 44639 records identified for import. Proceeding will save all imported records in the WeeWX archive. Are you sure you want to proceed (y/n)? y Unique records processed: 44639; Last timestamp: 2020-12-31 23:59:00 AEST (1609423140) Mapped data saved to archive successfully for period 1. Calculating missing derived observations ... Processing record: 44639; Last record: 2021-01-01 00:00:00 AEST (1609423200) Recalculating daily summaries... Records processed: 44000; time: 2020-12-31 13:20:00 AEST (1609384800) Finished recalculating daily summaries Finished calculating missing derived observations Finished import 44639 records were processed and 44639 unique records imported in 249.23 seconds. Those records with a timestamp already in the archive will not have been imported. Confirm successful import in the WeeWX log file. So I am confident there is nothing wrong with the import config file and log files posted. Granted it was a setup.py install but should not really make a difference to wee_import. You said you had 15 years of WD data, when you did the import were there other older log files in /tmp/wdisplay? The reason I ask is that whilst you can limit the time frame over which wee_import imports data, all that does is apply a filter to the final processing/saving of records. wee_import will still go through and read all log files it finds starting at the oldest. So if you had 15 years of WD data in a directory and tried to import the most recent month all 15 years of log files will be read and processed in a limited manner. The initial processing of the raw import data involves assembling the individual date and time components from each row in the log. That appears to be where your import failed. When I wrote the WD component of wee_import I found a number of data inconsistencies in the WD log files I was using so I guess it is possible you may have similar inconsistencies. Have a look through the oldest WD log file set in /tmp/wdisplay and see if you can spot an obvious error/inconsistency in the date and time data. If you can't spot anything could you post the respective log files here. Alternatively, if you let me know what version of WeeWX you are running I can provide you with an instrumented version of wee_import to log details of the offending record/log. Gary On Monday, 1 February 2021 at 00:59:35 UTC+10 peter.su...@gmail.com wrote: > Hi Gary, > > thanks for your quick answer. > Please check the attached file. All I edited was the path to data and > filenames. I limited it to only one type of file, with my last change. > > No, problem when it takes time. > Thanks for checking out > > Peter > gjr80 schrieb am Sonntag, 31. Januar 2021 um 13:58:21 UTC+1: > >> Hi, >> >> Nothing leaps out at me having a quick look at the output and log files >> you posted. I'll endeavour to have a look at this tomorrow (bit late here). >> Could I ask you to post the import config file you are using, >> /etc/weewx/wd-import-test.conf. There should be nothing sensitive in the >> file. >> >> Gary >> >> On Sunday, 31 January 2021 at 22:28:32 UTC+10 peter.su...@gmail.com >> wrote: >> >>> Hello, >>> after using my Vantage pro 2 with Weather Display for about 15 years, I >>> think now its time for a change. :) >>> I am making the first tests for transferring data, before switch the >>> live system. >>> >>> My problem, I cannot import the log data from wdisplay. >>> I'm getting this error message: >>> >>> pi@WEEWX-DEMO:/etc/weewx $ w
[weewx-user] Re: wee_import from WD logs fails
Hi Gary, thanks for your quick answer. Please check the attached file. All I edited was the path to data and filenames. I limited it to only one type of file, with my last change. No, problem when it takes time. Thanks for checking out Peter gjr80 schrieb am Sonntag, 31. Januar 2021 um 13:58:21 UTC+1: > Hi, > > Nothing leaps out at me having a quick look at the output and log files > you posted. I'll endeavour to have a look at this tomorrow (bit late here). > Could I ask you to post the import config file you are using, > /etc/weewx/wd-import-test.conf. There should be nothing sensitive in the > file. > > Gary > > On Sunday, 31 January 2021 at 22:28:32 UTC+10 peter.su...@gmail.com wrote: > >> Hello, >> after using my Vantage pro 2 with Weather Display for about 15 years, I >> think now its time for a change. :) >> I am making the first tests for transferring data, before switch the live >> system. >> >> My problem, I cannot import the log data from wdisplay. >> I'm getting this error message: >> >> pi@WEEWX-DEMO:/etc/weewx $ wee_import --import-config=wd-import-test.conf >> --from=2020-12-01T00:00 --to=2020-12-31T23:59 --verbose >> Using WeeWX configuration file /etc/weewx/weewx.conf >> Starting wee_import... >> Weather Display monthly log files in the '/tmp/wdisplay' directory will >> be imported >> The following options will be used: >> config=/etc/weewx/weewx.conf, >> import-config=/etc/weewx/wd-import-test.conf >> from=2020-12-01T00:00, to=2020-12-31T23:59 >> dry-run=None, calc_missing=True, ignore_invalid_data=True >> monthly logs are in METRIC units >> tranche=250, interval=1 >> UV=True, radiation=True ignore extreme temperature and humidity=True >> Using database binding 'wx_binding', which is bound to database >> 'weewx.sdb' >> Destination table 'archive' unit system is '0x01' (US). >> Missing derived observations will be calculated. >> Observations timestamped after 2020-12-01 00:00:00 CET (1606777200) and >> up to and >> including 2020-12-31 23:59:00 CET (1609455540) will be imported. >> Starting import ... >> Records covering multiple periods have been identified for import. >> Period 1 ... >> Obtaining raw import data for period 1 ... >> Traceback (most recent call last): >> File "/usr/share/weewx/wee_import", line 903, in >> main() >> File "/usr/share/weewx/wee_import", line 833, in main >> source_obj.run() >> File "/usr/share/weewx/weeimport/weeimport.py", line 369, in run >> _raw_data = self.getRawData(period) >> File "/usr/share/weewx/weeimport/wdimport.py", line 598, in getRawData >> _ts = "%s" % int(time.mktime(_tt)) >> OverflowError: mktime argument out of range >> >> >> Any ideas? File content is looking ok. >> It is happening with all time tests across this 15 years. >> Here a month sample data. >> >> Thanks for any ideas... >> Peter >> >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/fc65d014-fec6-4b8b-b1cf-fc6b150ebec9n%40googlegroups.com. # EXAMPLE CONFIGURATION FILE FOR IMPORTING FROM WEATHER DISPLAY (WD) # # Copyright (c) 2009-2019 Tom Keffer and Gary Roderick. # See the file LICENSE.txt for your rights. ## # Specify our source. Available options are: # CSV - import obs from a single CSV format file # WU - import obs from a Weather Underground PWS history # Cumulus - import obs from a one or more Cumulus monthly log files # WD - import obs from a one or more WD monthly log files # Format is: # source = (CSV | WU | Cumulus | WD) source = WD ## [WD] # Parameters used when importing WD monthly log files # # Directory containing WD monthly log files to be imported. Format is: # directory = full path without trailing / directory = /tmp/wdisplay # When importing WD monthly log file data the following WeeWX database # fields will be populated directly by the imported data: # barometer # dateTime # dewpoint # extraHumid1 (if MMYYYvantageextrasensorslog.csv is available and field hum1 has data) # extraHumid2 (if MMYYYvantageextrasensorslog.csv is available and field hum2 has data) # extraTemp1 (if MMYYYvantageextrasensorslog.csv is available and field temp1 has data) # extraTemp2 (if MMYYYvantageextrasensorslog.csv is available and field temp2 has data) # extraTemp3 (if MMYYYvantageextrasensorslog.csv is available and field temp3 has data) # heatindex # outHumidity # outTemp # radiation (if MMvantagelog.txt or MMvantagelogcsv.csv is available) # rain # soilTemp1
[weewx-user] Re: wee_import from WD logs fails
Hi, Nothing leaps out at me having a quick look at the output and log files you posted. I'll endeavour to have a look at this tomorrow (bit late here). Could I ask you to post the import config file you are using, /etc/weewx/wd-import-test.conf. There should be nothing sensitive in the file. Gary On Sunday, 31 January 2021 at 22:28:32 UTC+10 peter.su...@gmail.com wrote: > Hello, > after using my Vantage pro 2 with Weather Display for about 15 years, I > think now its time for a change. :) > I am making the first tests for transferring data, before switch the live > system. > > My problem, I cannot import the log data from wdisplay. > I'm getting this error message: > > pi@WEEWX-DEMO:/etc/weewx $ wee_import --import-config=wd-import-test.conf > --from=2020-12-01T00:00 --to=2020-12-31T23:59 --verbose > Using WeeWX configuration file /etc/weewx/weewx.conf > Starting wee_import... > Weather Display monthly log files in the '/tmp/wdisplay' directory will be > imported > The following options will be used: > config=/etc/weewx/weewx.conf, > import-config=/etc/weewx/wd-import-test.conf > from=2020-12-01T00:00, to=2020-12-31T23:59 > dry-run=None, calc_missing=True, ignore_invalid_data=True > monthly logs are in METRIC units > tranche=250, interval=1 > UV=True, radiation=True ignore extreme temperature and humidity=True > Using database binding 'wx_binding', which is bound to database 'weewx.sdb' > Destination table 'archive' unit system is '0x01' (US). > Missing derived observations will be calculated. > Observations timestamped after 2020-12-01 00:00:00 CET (1606777200) and up > to and > including 2020-12-31 23:59:00 CET (1609455540) will be imported. > Starting import ... > Records covering multiple periods have been identified for import. > Period 1 ... > Obtaining raw import data for period 1 ... > Traceback (most recent call last): > File "/usr/share/weewx/wee_import", line 903, in > main() > File "/usr/share/weewx/wee_import", line 833, in main > source_obj.run() > File "/usr/share/weewx/weeimport/weeimport.py", line 369, in run > _raw_data = self.getRawData(period) > File "/usr/share/weewx/weeimport/wdimport.py", line 598, in getRawData > _ts = "%s" % int(time.mktime(_tt)) > OverflowError: mktime argument out of range > > > Any ideas? File content is looking ok. > It is happening with all time tests across this 15 years. > Here a month sample data. > > Thanks for any ideas... > Peter > > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6e6c68b1-aafc-4ede-a8ee-1e02d8c3cbb6n%40googlegroups.com.
