After many, many hours the wee-import processed all but a handful of files. Have several that choked on the import. Ended up with 3,873,678 records.
Next I need to validate that the data imported to look for any potential bad data. On Friday, August 21, 2020 at 7:31:36 PM UTC-4 Andrew M wrote: > Thank you for the response. > > I actually changed up how I am going about this. My WD ran on a Windows10 > box. I had a RaspPi box sitting in a box so i decided to use that for WeeWx. > I was formatting the hard drive on the Win10 machine when i had the > thought that I should just put Debian on that and use that for WeeWx. > Which I did. This machine is a little faster that the RaspPi , so one I > have other things straight I will use that for the conversion. Migrating > the MySql from WD to the tables that WeeWX has is probably doable, but > really don't want to sit down and think about it that much. I will just let > the current weather collect on the RaspPi box and then once the import is > done on the now Debian box I can combine the two DB much easier. > > I have 60k rows on the MySQL DB with my current webhost, but more than > that saved external to that on a hard drive from a previous webhost that I > never moved. > > > > On Wednesday, August 19, 2020 at 8:37:29 PM UTC-4 gjr80 wrote: > >> Sorry can't help you with MySQL to MySQL migration. >> >> Regards wee_import though, yes it can be slow. When I wrote the WD >> import module the WD user who first used it in anger had something like >> (from memory) 10 years of data to import. The import had to be done in >> batches (again from memory) of 2-3 years each (wee_import uses >> transactions on the database but does keep track of duplicate timestamps >> and a few other things so memory usage does grow as the span of the import >> grows). I found one email from the user with the results of the first batch >> import and 1.4 millions records were imported on a Raspberry Pi in 58 >> minutes. Al things considered I find that reasonable. >> >> There are a couple of things you can do to speed up wee_import. You can >> tweak a the tranche setting in the import config file, this alters the >> size of the transactions (in records) that wee_import uses. The default is >> 250, you could raise this which will result in fewer db transactions but it >> will likely increase memory usage so you may need to do the import in >> smaller batches. One other approach if using a slow(ish) RPi as your WeeWX >> machine is to do just the import on a faster machine and then copy the >> imported data to the WeeWX RPi. Granted this is simpler when using SQLite >> but depending on your setup could be adapted for MySQL. >> >> You say you have 60 000 odd MySQL records, that does not seem like much, >> how does that correlate with the number of entries in the WD log files? >> >> Gary >> >> On Thursday, 20 August 2020 08:48:46 UTC+10, Andrew M wrote: >>> >>> I started to use the wee_import process to process all my WD log files >>> to WeeWx MySqlDB, and it was taking a long time. It seems like If I have >>> many years of data it will take that long to import them into WeeWx MySqlDB. >>> >>> I then thought, oh wait, i already have the WD data in a MySql DB, so >>> why am i doing this process. >>> >>> Now I have to figure out how I can gracefully import all the WD data I >>> have in a MySQL DB to the one I set up for WeeWx. Both are on the same >>> hosted server. Different DB names. >>> >>> There is one table for WD and multiple tables for WeeWx so have no idea >>> on where to begin with this. I have ~60,000 rows of data in the WD MySQL DB >>> table. >>> >>> Does anyone have a graceful way of migrating the WD MySQL DB into a >>> WeeWX DB? >>> >>> Am I overlooking something in the documentation or in group? >>> >>> Thank you. >>> >>> >>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2c9ffe0b-6490-466a-a807-decf40a04d68n%40googlegroups.com.
