When importing records with wee_import primary key duplicate entry log entries indicate a record with the timestamp of the current imported record already exists in the database. This most commonly occurs if the import period overlaps with existing database data. This is normal and not considered an error. In such cases the imported record is discarded and no database changes are made. If the imported record is required rather than the corresponding record in the database, the database record must be deleted and the import performed again. If the database record is to be retained then nothing need be done.
Gary On Tuesday, 31 October 2023 at 04:50:45 UTC+10 sylvain...@gmail.com wrote: > Hello Gary. > > Thank you very much for your work ! > I updated the weeimport.py file at 6:50 p.m. > > I created a small bash script (I'm not a programmer) to download my WU > data every hour (I didn't dare to be more greedy). The script is triggered > by the crontab. > My script retrieves the last timestamp from the mysql database and uses it > as dateFrom=%Y-%m-%dT%H:%M. The recovery date is dateTo=%Y-%m-%dT%H:%M. In > principle everything works well. > > If I understood correctly, the calculation error that you detected can > explain my mysql errors of this type 1062, "Duplicate entry '1698663292' > for key 'PRIMARY": > > #* journalctl -u weewx.service -p err -b -f --since today* > > > > > > > > > > > > > > > > > > > > > > > > > *...Oct 30 17:28:01 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 03:55:00 CEST (1657590900) > to database 'weewx': (1062, " Duplicate entry '1657590900' for key > 'PRIMARY'")Oct 30 17:33:02 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 04:00:00 CEST (1657591200) > to database 'weewx': (1062, " Duplicate entry '1657591200' for key > 'PRIMARY'")Oct 30 17:38:03 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 04:05:00 CEST (1657591500) > to database 'weewx': (1062, " Duplicate entry '1657591500' for key > 'PRIMARY'")Oct 30 17:43:04 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 04:10:00 CEST (1657591800) > to database 'weewx': (1062, " Duplicate entry '1657591800' for key > 'PRIMARY'")Oct 30 17:48:04 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 04:15:00 CEST (1657592100) > to database 'weewx': (1062, " Duplicate entry '1657592100' for key > 'PRIMARY'")Oct 30 17:53:04 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 04:20:00 CEST (1657592400) > to database 'weewx': (1062, " Duplicate entry '1657592400' for key > 'PRIMARY'")Oct 30 17:58:05 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 04:25:00 CEST (1657592700) > to database 'weewx': (1062, " Duplicate entry '1657592700' for key > 'PRIMARY'")Oct 30 18:03:05 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 04:30:00 CEST (1657593000) > to database 'weewx': (1062, " Duplicate entry '1657593000' for key > 'PRIMARY'")Oct 30 18:08:05 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 04:35:00 CEST (1657593300) > to database 'weewx': (1062, " Duplicate entry '1657593300' for key > 'PRIMARY'")Oct 30 18:13:05 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 04:40:00 CEST (1657593600) > to database 'weewx': (1062, " Duplicate entry '1657593600' for key > 'PRIMARY'")Oct 30 18:18:07 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 04:45:00 CEST (1657593900) > to database 'weewx': (1062, " Duplicate entry '1657593900' for key > 'PRIMARY'")Oct 30 18:23:07 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 04:50:00 CEST (1657594200) > to database 'weewx': (1062, " Duplicate entry '1657594200' for key > 'PRIMARY'")Oct 30 18:28:07 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 04:55:00 CEST (1657594500) > to database 'weewx': (1062, " Duplicate entry '1657594500' for key > 'PRIMARY'")Oct 30 18:33:07 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 05:00:00 CEST (1657594800) > to database 'weewx': (1062, " Duplicate entry '1657594800' for key > 'PRIMARY'")Oct 30 18:38:08 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 05:05:00 CEST (1657595100) > to database 'weewx': (1062, " Duplicate entry '1657595100' for key > 'PRIMARY'")Oct 30 18:43:08 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 05:10:00 CEST (1657595400) > to database 'weewx': (1062, " Duplicate entry '1657595400' for key > 'PRIMARY'")Oct 30 18:48:08 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 05:15:00 CEST (1657595700) > to database 'weewx': (1062, " Duplicate entry '1657595700' for key > 'PRIMARY'")Oct 30 18:53:08 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 05:20:00 CEST (1657596000) > to database 'weewx': (1062, " Duplicate entry '1657596000' for key > 'PRIMARY'")Oct 30 18:58:09 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 05:25:00 CEST (1657596300) > to database 'weewx': (1062, " Duplicate entry '1657596300' for key > 'PRIMARY'")Oct 30 19:03:09 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 05:30:00 CEST (1657596600) > to database 'weewx': (1062, " Duplicate entry '1657596600' for key > 'PRIMARY'")Oct 30 19:08:09 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 05:35:00 CEST (1657596900) > to database 'weewx': (1062, "Duplicate entry '1657596900' for key > 'PRIMARY'")Oct 30 19:13:09 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 05:40:00 CEST (1657597200) > to database 'weewx': (1062, " Duplicate entry '1657597200' for key > 'PRIMARY'")Oct 30 19:18:10 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 05:45:00 CEST (1657597500) > to database 'weewx': (1062, " Duplicate entry '1657597500' for key > 'PRIMARY'")Oct 30 19:23:10 gandalf python3[1261]: weewx[1261] ERROR > weewx.manager: Unable to add record 2022-07-12 05:50:00 CEST (1657597800) > to database 'weewx': (1062, " Duplicate entry '1657597800' for key > 'PRIMARY'")* > > > I don't understand why there are Duplicate entry errors on different keys? > > Do you have any idea what's going on? > > > Le lundi 30 octobre 2023 à 13:42:32 UTC+1, gjr80 a écrit : > >> I have been able to reproduce your symptoms and I now believe I >> understand what is happening. >> >> In terms of why wee_import wants to import from 27 and 28 October Vince >> has covered the main point. So as far as WeeWX is concerned (and assuming a >> five minute archive period) the first archive record of the 27 October >> covers the five minute period from midnight at the start of 27 October >> until 00:05, so the first archive period of the day is timestamped at five >> minutes after midnight. The last archive period covers the five minute >> period from 23:55 until (and including) midnight at the end of 27 October, >> so the last archive period of 27 October is timestamped at midnight on 28 >> October. >> >> The other factor at play is how WU provides data via its API. The WU API >> call used by wee_import provides one full day of data at a time, the >> problem being that WU defines a day as starting at midnight at the start of >> the day though until (but not including) midnight at the end of the day. So >> when the WU API responds with data for 27 October the first record provided >> is the archive record timestamped at midnight at the start of 27 October, >> this archive record actually contains data from 26 October so it is >> discarded. The last record provided by WU is timestamped at 23:55 27 >> October, which is only the second last archive record for the day. WU does >> not provide the last archive record of the day, to get that archive record >> we need to ask WU for the data for 28 October. The first record from the WU >> 28 October data is used and the rest are discarded. >> >> That is why when you attempt to import a given day of data from WU you >> will actually see wee_import obtain two days data from WU. >> >> Now for your error message. To cut a long story short there was a flaw in >> how wee_import calculated the total elapsed time for an import. I >> believe I have fixed it. You can use the new code by downloading the >> updated weeimport.py and replacing weeimport.py on your system with the >> downloaded version. In an earlier post I gave you instructions for >> replacing wuimport.py, this time you need to replace weeimport.py, it is >> in the same directory as wuimport.py and the process is the same just a >> different file. You can download the updated weeimport.py via the >> following command: >> >> $ wget -P /var/tmp >> https://raw.githubusercontent.com/weewx/weewx/V5/src/weeimport/weeimport.py >> >> Do not get weeimport.py confused with wee_import.py, they are in >> different directories but if you replace the wrong file you will break >> wee_import. >> >> Once you have replaced weeimport.py wee_import should work without error. >> >> Gary >> >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/62a01952-e91e-493f-a6fd-4b8b09932c03n%40googlegroups.com.
