Peter,
I ran wee_import under WeeWX v4.4.0/python3 using the import config file
and log files posted and the import processed without error:
gary@buster31:~$ /home/weewx/bin/wee_import
--import-config=/var/tmp/wd-import-test.conf --from=2020-12-01T00:00
--to=2020-12-31T23:59 --verbose
Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Weather Display monthly log files in the '/tmp/wdisplay' directory will be
imported
The following options will be used:
config=/home/weewx/weewx.conf,
import-config=/var/tmp/wd-import-test.conf
from=2020-12-01T00:00, to=2020-12-31T23:59
dry-run=None, calc_missing=True, ignore_invalid_data=True
monthly logs are in METRIC units
tranche=250, interval=1
UV=True, radiation=True ignore extreme temperature and humidity=True
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Observations timestamped after 2020-12-01 00:00:00 AEST (1606744800) and up
to and
including 2020-12-31 23:59:00 AEST (1609423140) will be imported.
Starting import ...
Records covering multiple periods have been identified for import.
Obtaining raw import data for period 1 ...
Raw import data read successfully for period 1.
Mapping raw import data for period 1 ...
Mapped 44639 records.
Raw import data mapped successfully for period 1.
Saving mapped data to archive for period 1 ...
44639 records identified for import.
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)? y
Unique records processed: 44639; Last timestamp: 2020-12-31 23:59:00 AEST
(1609423140)
Mapped data saved to archive successfully for period 1.
Calculating missing derived observations ...
Processing record: 44639; Last record: 2021-01-01 00:00:00 AEST (1609423200)
Recalculating daily summaries...
Records processed: 44000; time: 2020-12-31 13:20:00 AEST (1609384800)
Finished recalculating daily summaries
Finished calculating missing derived observations
Finished import
44639 records were processed and 44639 unique records imported in 249.23
seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
So I am confident there is nothing wrong with the import config file and
log files posted. Granted it was a setup.py install but should not really
make a difference to wee_import.
You said you had 15 years of WD data, when you did the import were there
other older log files in /tmp/wdisplay? The reason I ask is that whilst you
can limit the time frame over which wee_import imports data, all that does
is apply a filter to the final processing/saving of records. wee_import
will still go through and read all log files it finds starting at the
oldest. So if you had 15 years of WD data in a directory and tried to
import the most recent month all 15 years of log files will be read and
processed in a limited manner. The initial processing of the raw import
data involves assembling the individual date and time components from each
row in the log. That appears to be where your import failed. When I wrote
the WD component of wee_import I found a number of data inconsistencies in
the WD log files I was using so I guess it is possible you may have similar
inconsistencies. Have a look through the oldest WD log file set in
/tmp/wdisplay and see if you can spot an obvious error/inconsistency in the
date and time data. If you can't spot anything could you post the
respective log files here. Alternatively, if you let me know what version
of WeeWX you are running I can provide you with an instrumented version of
wee_import to log details of the offending record/log.
Gary
On Monday, 1 February 2021 at 00:59:35 UTC+10 peter.su...@gmail.com wrote:
> Hi Gary,
>
> thanks for your quick answer.
> Please check the attached file. All I edited was the path to data and
> filenames. I limited it to only one type of file, with my last change.
>
> No, problem when it takes time.
> Thanks for checking out
>
> Peter
> gjr80 schrieb am Sonntag, 31. Januar 2021 um 13:58:21 UTC+1:
>
>> Hi,
>>
>> Nothing leaps out at me having a quick look at the output and log files
>> you posted. I'll endeavour to have a look at this tomorrow (bit late here).
>> Could I ask you to post the import config file you are using,
>> /etc/weewx/wd-import-test.conf. There should be nothing sensitive in the
>> file.
>>
>> Gary
>>
>> On Sunday, 31 January 2021 at 22:28:32 UTC+10 peter.su...@gmail.com
>> wrote:
>>
>>> Hello,
>>> after using my Vantage pro 2 with Weather Display for about 15 years, I
>>> think now its time for a change. :)
>>> I am making the first tests for transferring data, before switch the
>>> live system.
>>>
>>> My problem, I cannot import the log data from wdisplay.
>>> I'm getting this error message:
>>>
>>> pi@WEEWX-DEMO:/etc/weewx $ wee_import
>>> --import-config=wd-import-test.conf --from=2020-12-01T00:00
>>> --to=2020-12-31T23:59 --verbose
>>> Using WeeWX configuration file /etc/weewx/weewx.conf
>>> Starting wee_import...
>>> Weather Display monthly log files in the '/tmp/wdisplay' directory will
>>> be imported
>>> The following options will be used:
>>> config=/etc/weewx/weewx.conf,
>>> import-config=/etc/weewx/wd-import-test.conf
>>> from=2020-12-01T00:00, to=2020-12-31T23:59
>>> dry-run=None, calc_missing=True, ignore_invalid_data=True
>>> monthly logs are in METRIC units
>>> tranche=250, interval=1
>>> UV=True, radiation=True ignore extreme temperature and humidity=True
>>> Using database binding 'wx_binding', which is bound to database
>>> 'weewx.sdb'
>>> Destination table 'archive' unit system is '0x01' (US).
>>> Missing derived observations will be calculated.
>>> Observations timestamped after 2020-12-01 00:00:00 CET (1606777200) and
>>> up to and
>>> including 2020-12-31 23:59:00 CET (1609455540) will be imported.
>>> Starting import ...
>>> Records covering multiple periods have been identified for import.
>>> Period 1 ...
>>> Obtaining raw import data for period 1 ...
>>> Traceback (most recent call last):
>>> File "/usr/share/weewx/wee_import", line 903, in <module>
>>> main()
>>> File "/usr/share/weewx/wee_import", line 833, in main
>>> source_obj.run()
>>> File "/usr/share/weewx/weeimport/weeimport.py", line 369, in run
>>> _raw_data = self.getRawData(period)
>>> File "/usr/share/weewx/weeimport/wdimport.py", line 598, in getRawData
>>> _ts = "%s" % int(time.mktime(_tt))
>>> OverflowError: mktime argument out of range
>>>
>>>
>>> Any ideas? File content is looking ok.
>>> It is happening with all time tests across this 15 years.
>>> Here a month sample data.
>>>
>>> Thanks for any ideas...
>>> Peter
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/11eed5a9-de66-4576-9803-e322a9bb24b6n%40googlegroups.com.
