Thank you for the response. I actually changed up how I am going about this. My WD ran on a Windows10 box. I had a RaspPi box sitting in a box so i decided to use that for WeeWx. I was formatting the hard drive on the Win10 machine when i had the thought that I should just put Debian on that and use that for WeeWx. Which I did. This machine is a little faster that the RaspPi , so one I have other things straight I will use that for the conversion. Migrating the MySql from WD to the tables that WeeWX has is probably doable, but really don't want to sit down and think about it that much. I will just let the current weather collect on the RaspPi box and then once the import is done on the now Debian box I can combine the two DB much easier.
I have 60k rows on the MySQL DB with my current webhost, but more than that saved external to that on a hard drive from a previous webhost that I never moved. On Wednesday, August 19, 2020 at 8:37:29 PM UTC-4 gjr80 wrote: > Sorry can't help you with MySQL to MySQL migration. > > Regards wee_import though, yes it can be slow. When I wrote the WD import > module the WD user who first used it in anger had something like (from > memory) 10 years of data to import. The import had to be done in batches > (again from memory) of 2-3 years each (wee_import uses transactions on > the database but does keep track of duplicate timestamps and a few other > things so memory usage does grow as the span of the import grows). I found > one email from the user with the results of the first batch import and 1.4 > millions records were imported on a Raspberry Pi in 58 minutes. Al things > considered I find that reasonable. > > There are a couple of things you can do to speed up wee_import. You can > tweak a the tranche setting in the import config file, this alters the > size of the transactions (in records) that wee_import uses. The default is > 250, you could raise this which will result in fewer db transactions but it > will likely increase memory usage so you may need to do the import in > smaller batches. One other approach if using a slow(ish) RPi as your WeeWX > machine is to do just the import on a faster machine and then copy the > imported data to the WeeWX RPi. Granted this is simpler when using SQLite > but depending on your setup could be adapted for MySQL. > > You say you have 60 000 odd MySQL records, that does not seem like much, > how does that correlate with the number of entries in the WD log files? > > Gary > > On Thursday, 20 August 2020 08:48:46 UTC+10, Andrew M wrote: >> >> I started to use the wee_import process to process all my WD log files to >> WeeWx MySqlDB, and it was taking a long time. It seems like If I have many >> years of data it will take that long to import them into WeeWx MySqlDB. >> >> I then thought, oh wait, i already have the WD data in a MySql DB, so why >> am i doing this process. >> >> Now I have to figure out how I can gracefully import all the WD data I >> have in a MySQL DB to the one I set up for WeeWx. Both are on the same >> hosted server. Different DB names. >> >> There is one table for WD and multiple tables for WeeWx so have no idea >> on where to begin with this. I have ~60,000 rows of data in the WD MySQL DB >> table. >> >> Does anyone have a graceful way of migrating the WD MySQL DB into a WeeWX >> DB? >> >> Am I overlooking something in the documentation or in group? >> >> Thank you. >> >> >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/98470bfb-98f7-440f-92ea-d9daebd15e87n%40googlegroups.com.
