Hi, Will be cool if you create a package with this script and with instructions to patch ZODB and maybe new major version could have the new API.
If you need help I'm interested to do it. :-) Cheers, On Thu, Feb 14, 2013 at 6:29 PM, Jürgen Herrmann <juergen.herrm...@xlhost.de > wrote: > Am 07.02.2013 17:11, schrieb Jim Fulton: > >> On Thu, Feb 7, 2013 at 10:48 AM, Jürgen Herrmann >> >> <juergen.herrm...@xlhost.de> wrote: >> >>> Am 06.02.2013 15:05, schrieb Jürgen Herrmann: >>> >>> Hi there! >>>> >>>> I hav a relstorage with mysql backend that grew out of bounds >>>> and we're looking into different backend solutions now. Possibly >>>> also going back to FileStorage and using zeo... >>>> >>>> Anyway we'll have to convert the databases at some point. As these >>>> are live DBs we cannot shut them down for longer than the >>>> ususal maintenance interval during the night, so for maybe 2-3h. >>>> >>>> a full conversion process will never complete in this time so >>>> we're looking for a process that can split the conversion into >>>> two phases: >>>> >>>> 1. copy transactions from backup of the source db to the destination >>>> db. this can take a long time, we don't care. note the last >>>> timestamp/transaction_id converted. >>>> 2. shut down the source db >>>> 3. copy transactions from the source db to the destination db, starting >>>> at the last converted transaction_id. this should be fast, as only >>>> a few transactions need to be converted, say < 1% . >>>> >>>> >>>> if i would reimplement copyTransactionsFrom() to accept a start >>>> transaction_id/timestamp, would this result in dest being an exact >>>> copy of source? >>>> >>>> source = open_my_source_storage() >>>> dest = open_my_destination_storage() >>>> dest.copyTransactionsFrom(**source) >>>> last_txn_id = source.lastTransaction() >>>> source.close() >>>> dest.close() >>>> >>>> source = open_my_source_storage() >>>> # add some transactions >>>> source.close() >>>> >>>> source = open_my_source_storage() >>>> dest = open_my_destination_storage() >>>> dest.copyTransactionsFrom(**source, last_txn_id=last_txn_id) >>>> source.close() >>>> dest.close() >>>> >>> >>> >>> I will reply to myself here :) This actually works, tested with a >>> modified version of FileStorage for now. I modified the signature >>> of copyTransactionsFrom to look like this: >>> >>> def copyTransactionsFrom(self, source, verbose=0, not_before_tid=None): >>> >> >> ``start`` would be better to be consistent with the iterator API. >> >> not_before_tid is a packed tid or None, None meaning "copy all" >>> (the default, so no existing API usage would break). >>> >>> Is there public interest in modifying this API permamently? >>> >> >> +.1 >> >> This "API" is a bit of an attractive nuisance. I'd rather people >> learn how to use iterators in their own scripts, as they are very >> useful and powerful. This API just hides that. >> >> The second part, replaying old transactions is a bit more subtle, >> but it's still worth it for people to be aware of it. >> >> If I were doing this today, I'd make this documentation >> rather than API. But then, documentation ... whimper. >> >> Anybody want to look at the actual code changes? >>> >> >> Sure, if they have tests. Unfortunately, we can only accept >> pull requests from zope contributors. Are you one? >> Wanna be one? :) >> >> Jim >> > > 1. SUCCESS. I migrated to FileStorage/ZEO successfully with the modified > copyTransactionsFrom() with downtime of only 10 minutes. Very cool :) > DB sizes are reasonable again and repozo backups are set up, just like > before the migration to RelStorage. Feels robust again, a good feeling. > I'd advise anybody thinking about migrating to RelStorage to think > about a proper (incremental) backup strategy for the underlying sql db > first, this might become crucial! repozo makes your life easy there > when using FileStorage. > > 2. Regarding the tests: I did not find any test for copyTransactionsFrom() > in the current test suite, am I blind? If there isn't any test yet, > maybe you could suggest a proper file for the test, maybe even a similar > one to start from? > > 3. Regarding my request to have this in ZODB 3.10.5: > After reading my own statement I quickly realized that changing an API > can only happen in a major version. Anyway, this was a one-shot action > so I'll probably never need my own code again :) > > best regards, > Jürgen > > -- > >> XLhost.de ® - Webhosting von supersmall bis eXtra Large << >>> >> > XLhost.de GmbH > Jürgen Herrmann, Geschäftsführer > Boelckestrasse 21, 93051 Regensburg, Germany > > Geschäftsführer: Jürgen Herrmann > Registriert unter: HRB9918 > Umsatzsteuer-**Identifikationsnummer: DE245931218 > > Fon: +49 (0)800 XLHOSTDE [0800 95467833] > Fax: +49 (0)800 95467830 > Web: http://www.XLhost.de > ______________________________**_________________ > For more information about ZODB, see http://zodb.org/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > https://mail.zope.org/mailman/**listinfo/zodb-dev<https://mail.zope.org/mailman/listinfo/zodb-dev> > -- Rudá Porto Filgueiras http://python-blog.blogspot.com http://twitter.com/rudaporto
_______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
