If you to provide some details on problems the handful of files experienced I am happy to look at wee_import to see if any changes can be made to improve its handling of such files. The WD import module of wee_import was developed based on a handful of WD log files found on the internet, so it is quite possible there are some corner cases that may cause wee_import to reject a file.
Gary On Wednesday, 2 September 2020 at 07:08:20 UTC+10 Andrew M wrote: > After many, many hours the wee-import processed all but a handful > of files. Have several that choked on the import. > Ended up with 3,873,678 records. > > Next I need to validate that the data imported to look for any potential > bad data. > > > > On Friday, August 21, 2020 at 7:31:36 PM UTC-4 Andrew M wrote: > >> Thank you for the response. >> >> I actually changed up how I am going about this. My WD ran on a Windows10 >> box. I had a RaspPi box sitting in a box so i decided to use that for WeeWx. >> I was formatting the hard drive on the Win10 machine when i had the >> thought that I should just put Debian on that and use that for WeeWx. >> Which I did. This machine is a little faster that the RaspPi , so one I >> have other things straight I will use that for the conversion. Migrating >> the MySql from WD to the tables that WeeWX has is probably doable, but >> really don't want to sit down and think about it that much. I will just let >> the current weather collect on the RaspPi box and then once the import is >> done on the now Debian box I can combine the two DB much easier. >> >> I have 60k rows on the MySQL DB with my current webhost, but more than >> that saved external to that on a hard drive from a previous webhost that I >> never moved. >> >> >> >> On Wednesday, August 19, 2020 at 8:37:29 PM UTC-4 gjr80 wrote: >> >>> Sorry can't help you with MySQL to MySQL migration. >>> >>> Regards wee_import though, yes it can be slow. When I wrote the WD >>> import module the WD user who first used it in anger had something like >>> (from memory) 10 years of data to import. The import had to be done in >>> batches (again from memory) of 2-3 years each (wee_import uses >>> transactions on the database but does keep track of duplicate timestamps >>> and a few other things so memory usage does grow as the span of the import >>> grows). I found one email from the user with the results of the first batch >>> import and 1.4 millions records were imported on a Raspberry Pi in 58 >>> minutes. Al things considered I find that reasonable. >>> >>> There are a couple of things you can do to speed up wee_import. You can >>> tweak a the tranche setting in the import config file, this alters the >>> size of the transactions (in records) that wee_import uses. The default is >>> 250, you could raise this which will result in fewer db transactions but it >>> will likely increase memory usage so you may need to do the import in >>> smaller batches. One other approach if using a slow(ish) RPi as your WeeWX >>> machine is to do just the import on a faster machine and then copy the >>> imported data to the WeeWX RPi. Granted this is simpler when using SQLite >>> but depending on your setup could be adapted for MySQL. >>> >>> You say you have 60 000 odd MySQL records, that does not seem like much, >>> how does that correlate with the number of entries in the WD log files? >>> >>> Gary >>> >>> On Thursday, 20 August 2020 08:48:46 UTC+10, Andrew M wrote: >>>> >>>> I started to use the wee_import process to process all my WD log files >>>> to WeeWx MySqlDB, and it was taking a long time. It seems like If I have >>>> many years of data it will take that long to import them into WeeWx >>>> MySqlDB. >>>> >>>> I then thought, oh wait, i already have the WD data in a MySql DB, so >>>> why am i doing this process. >>>> >>>> Now I have to figure out how I can gracefully import all the WD data I >>>> have in a MySQL DB to the one I set up for WeeWx. Both are on the same >>>> hosted server. Different DB names. >>>> >>>> There is one table for WD and multiple tables for WeeWx so have no >>>> idea on where to begin with this. I have ~60,000 rows of data in the WD >>>> MySQL DB table. >>>> >>>> Does anyone have a graceful way of migrating the WD MySQL DB into a >>>> WeeWX DB? >>>> >>>> Am I overlooking something in the documentation or in group? >>>> >>>> Thank you. >>>> >>>> >>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/4543ed66-c783-4f14-ab5f-0ca5fffc6177n%40googlegroups.com.
