Hi all, davidb started a branch of RelStorage in July 2010 and I wonder if that can be merged. It does two interesting things in the zodbconvert code:
- It adds a --single-transaction command line option. From the readme: "Import into the destination in a single transaction, instead of one transaction per transaction in the source. This option can significantly speed up conversion times on PostgreSQL." - I adds more logging so you have an indication that yes, something is in fact happening. This is good for your heart when you want to migrate a 16 GB Data.fs. :-) Otherwise you just see 'Storages opened successfully' and then nothing at all for several hours. I am no zodb or relstorage expert, but the changes for the --single-transaction code look good to me. In practice it seems to work too. I converted a CatalogData.fs of 150 MB to postgres. With --single-transaction the final result printed was: All 2635 transactions copied successfully in 4.3 minutes at 0.550mB/s. Without that option the result was: All 2635 transactions copied successfully in 6.2 minutes at 0.379mB/s. This is about a 30% decrease in time using the --single-transaction option. In both cases the target postgres database was already filled so I used the --clear option as well. The conversion was done on the server that had the CatalogData.fs. The postgres database was hosted on a different server in the same network. I added some more logging on that branch, mostly because the conversion appeared to be hanging at some unknown point. This was also with the official 1.4.1 release, which was the reason I started experimenting with the fastimport branch to see if that would help. It did not. At least in both test runs mentioned above, the actual time it took was about twenty minutes longer, possibly because the conversion temporarily lost the connection with the postgres server. With the logging I could at least see that it was throwing the old transaction table away; I have seen the same with other tables. Definitely no one else is accessing this database at the same time. So if someone has an idea what could be going on here, that is welcome. Anyway, the --single-transaction seems to work and I would say the logging is helpful. So: is there any chance this can be merged to trunk? I am also interested in the blob support that has been added there. :-) Thanks, -- Maurits van Rees Programmer, Zest Software _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - [email protected] https://mail.zope.org/mailman/listinfo/zodb-dev
