On Monday, 23 March 2020 02:09:52 UTC+10, Lucas Heijst wrote: > > > In my case I want to import a year of data! >
Given WU's history of flakiness and given that your year of data will require 350 odd WU API calls via the internet, I would not be doing this as one big import; you are just asking for a failure. I would do a month at a time starting at the oldest data and working through to the most recent. If a given month fails to import for some reason, you should be able to re-import the month without deleting any data (untested and I am relying on memory here). Of course if you have reason to believe that corrupt data was stored then you need delete that before re-importing. > If during this import (which will take very long) something goes wrong, > the program is aborted and no derived values will be calculated. > When I start the batch all over from the beginning, tjhe raw records which > are already in the database will be skipped (and not overwritten). > The derived values also won't be calculated for those existing records. > > If you look at the 4.0 Utilities Guide you will notice that the wee_database utility now has a --calc-missing action. This allows users to run the routine that calculates missing derived obs separately from wee_import. Note that the calc-missing routine will only update fields that are None or otherwise contain no data (hence the 'missing' in calc-missing), if a field contains data calc-missing will not change it. A beter approach would be to calculate the derived values per tranche > (default: 250 records). > Then when something goes wrong, we can start with the a new date (short > after the latest catched record). > If you have a look at issue #443 <https://github.com/weewx/weewx/issues/443> you will see that approach was considered but not possible due to some architectural issues. Hence the approach that was adopted that made the calc-missing routine separate to, but callable from wee_import. Undertaking imports in smaller chunks (especially from WU) and being able to separately call the calc-missing routine from wee_database help mitigate against errors during imports. > > Luc > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/6cdd49cd-97e3-4cf9-87d0-2883aaecae62%40googlegroups.com.
