Rich,
I sat down and imported the Sep17 log you posted under weeWX 3.8.0 and it
imported 3877 records without any error. The Cumulus monthly log file uses
a fixed format date and time fields as the first 2 fields of the line; the
formats are d/m/Y and H:M:S and this is hard coded into wee_import so there
is nothing that could be set incorrectly d/m wise.
In terms of the date-time order of what is being imported, wee_import
orders the Cumulus logs in oldest to newest order and then proceeds to work
throught each file from oldest to newest importing all data in each file
(which happens to be in oldest to newest order as well). wee_import
processes the imported records in tranches, nominally 250 records per
tranche. I don't see any way that data could be processed twice during an
import (it can certainly be processed twice if you import it twice but not
if you only import it once). All told I don't see how any of this could
have caused the errors you received. All in all I am at a bit of a loss.
I am wondering if somehow some of your data found its way into your
archive. Could I ask you to do a little interrogating of your archive?
Given there were issues with the September 2017 data and that you posted
the September 2017 log file lets have a look at that month. I'm assuming
you are using SQLite and you database is /home/weewx/archive/weewx.sdb,
open your archive with SQLite3:
$ sqlite3 /home/weewx/archive/weewx.sdb
sqlite>
Check how many records are in the month of Spetember 2017:
sqlite> SELECT COUNT(dateTime) FROM archive WHERE
dateTime>=strftime('%s','2017-09-01
00:00:00', 'utc') AND dateTime<strftime('%s','2017-10-01 00:00:00', 'utc');
That should return a number, there are 8377 valid records in your September
2017 monthly log file so if the answer is 8377 you have almost a full month
of data (there were a few records missing 30*288=8640). Assuming there is
almost full month of data in you archive lets pick a record from a period
that was not imported, say 20 September 2017 13:55 and see what it holds:
sqlite> SELECT outTemp, outHumidity, dewpoint FROM archive WHERE dateTime=
strftime('%s','2017-09-20 13:55:00', 'utc');
82.2|55.0|64.4
outTemp, outHumidity and dewpoint are the 3rd, 4th and 5th fields on each
line so easy to check. On your imported data I get 82.5F, 55.0% and 64.4F,
how does that compare with your results? If the same I strongly suspect
your data was somehow imported beforehand, you can check further
records/obs by changing the fields and date/time in the above query. If
your results don't agree it looks like you have some bogus data somehow. If
you have bogus data then I guess the only solution is to delete it and
re-import. That is easy to do for just one month, happy to give you the SQL
separately if required. I guess if your results agree with what's in you
monthly log file then you need to make a decision whether you trust you
have all of the correct data or not (do you look at other months/days/times
- there is a lot to look at), if you trust it then not much else to do but
continue importing your older data (I would import one month at a time and
check that there is no data in your archive for that month before you
import it, do the month import and see if it all imported OK or of you have
more errors, generally easier to pick uon on and fix an issue this way then
doing a multi-year import and then having mountains to deal with.).
One other thing you can check, the logs. After a wee_import run you should
have a summary written to log (unless you specifically turn it off), I
received something like:
Jan 29 21:05:39 stretch22 wee_import[1256]: manager: Added record 2017-09-30
23:55:00 AEST (1506779700) to database 'weewx.sdb'
Jan 29 21:05:39 stretch22 wee_import[1256]: manager: Added record 2017-09-30
23:55:00 AEST (1506779700) to daily summary in 'weewx.sdb'
Jan 29 21:05:39 stretch22 wee_import[1256]: wee_import: Mapped data saved
to archive successfully for period 1.
Jan 29 21:05:39 stretch22 wee_import[1256]: wee_import: Finished import.
8377 raw records resulted in 8377 unique records being processed in 25.55
seconds.
when I imported your September data. Worthwhile having at look at the
numbers to see how many duplicates you had (there were none for the above
import), should give an idea of the extent of the issue though not much
other help.
One final thing, if you do do any more imports keep a copy of the screen
output, it should be providing progress on what wee_import is doing as well
as the log.
Gary
On Monday, 29 January 2018 06:27:04 UTC+10, gjr80 wrote:
>
> I did see the first couple of posts this morning but it was way too early
> and I needed to get going. I will have a look later when home again, but my
> first thoughts were that
>
> manager: Unable to add record 2017-11-15 00:00:00 EST (1512018000) to
> database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
>
> indicates that a record of that timestamp already exists in the archive. I
> guess DD/MM MM/DD could cause this but if the date format is consistent
> across all source records/files it shouldn't.
>
> As for order of import, I thought wee_import ordered the Cumulus monthly
> log files from oldest to newest, though I suspect it would not take too
> much to upset this, it is only a rudimentary check/order based on file
> name. But as you say rebuilding the daily summaries will fix this.
>
> Gary
>
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
