[ZODB-Dev] ZODB replication and high-availability across data centers
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi there, we are currently investigating options for a large-scale solution where the machines (ZEO clients and ZEO server) should be separated across two data centers. Incoming traffic should be balanced between both data centers. Automatic failover and online-replication are desirable - "high-performance" is a must. The well-known options are: ZRS (ZEO) Relstorage on top of some RDBMS-related clustering/replication mechanism NEO DRDB ZEORaid Has anyone worked on such a setup or may share some experiences - especially by using one of the options above across data-centers (with a limited bandwidth inbetween and a higher latency compared to LAN). Andreas -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJMwF/VAAoJEADcfz7u4AZjTaQLvAxoZINqbokK2Zyeep0ARtNL /nIHfkyxkVWSqM+Tgdl7NaBvjwkkdx/+Dm4jl3F0w2NojYGRCDaHI44yCrMvfsJU fQD+tzJ8UDOEgbVYxAVMdzX6vMCJuzL9KkqK7X5Nzt53KKxV8wkuaAZExFJ6dzBR EHy0AzwYZQy90VEoX/xnnHZk3HJ5ufuX/AWMNUrxVfsAf9tN5/e0xSSJtQxG4nD5 yEfMrl7jmKNjYh4pjR2dV38htrCO1fWngVE1QjUwwHqj29G7ccXzFl/h1UYuPyrF q0TtrMQQzAcYh1vXm1DiZU0MKHHhQBhztgQteZUouzQpowpe7HODc8Z/h29wUtTn R92ErDM9wccgtpLjRZbvpLf3Uj0doDW0BX+PV/cfKFYutqUplSpMsGrrsWiO7XxO YulxDXNqKtkqeN7XncCLCnzPtnCreyjWeVBQ4cwWgA2HNggxDJ3G8/oqCNCK+tTW h4jgqZGtM3NHmMW8T63BvruRLj0RnFE= =DQIG -END PGP SIGNATURE- <>___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB replication and high-availability across data centers
On Thu, 2010-10-21 at 17:44 +0200, Andreas Jung wrote: > Hi there, > > we are currently investigating options for a large-scale solution > where the machines (ZEO clients and ZEO server) should be separated > across two data centers. Incoming traffic should be balanced between > both data centers. Automatic failover and online-replication are > desirable - "high-performance" is a must. > > The well-known options are: > > ZRS > (ZEO) > Relstorage on top of some RDBMS-related clustering/replication mechanism > NEO > DRDB > ZEORaid We use DRDB regularly for replication between servers but never across data centers. > > Has anyone worked on such a setup or may share some experiences - > especially by using one of the options above across data-centers > (with a limited bandwidth inbetween and a higher latency compared to LAN). We have only one application where we do multi-master replication between SA and Germany over a severely limited bandwidth connection. The replication is at the application level and not at the database level since we wanted to finely control what we transfer and ensure we transfer as little as possible. The replication code is a single Python module that serializes method calls on objects, transports it to the replication target and repeats the call on the replication target. It's modelled on the idea of replaying SQL statements on a replication target. Parts of it are application specific but most of it is pretty generic. If you or anybody else is interested I can mail it to you - I've never had the time to turn this into something generally useful. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB replication and high-availability across data centers
On 21/10/2010 16:44, Andreas Jung wrote: > The well-known options are: > > Relstorage on top of some RDBMS-related clustering/replication mechanism This is where I'd look, since the RDBMS guys likely have patterns that match the one you're seeking... > ZEORaid Not yet! (and, performance wise, I think CT would agree with me, that might well be "not ever") Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB replication and high-availability across data centers
> we are currently investigating options for a large-scale solution > where the machines (ZEO clients and ZEO server) should be separated > across two data centers. Incoming traffic should be balanced between > both data centers. Automatic failover and online-replication are > desirable - "high-performance" is a must. > > The well-known options are: > > ZRS > (ZEO) > Relstorage on top of some RDBMS-related clustering/replication mechanism > NEO > DRDB > ZEORaid We have not explicitly set something like this up; although our hosting partner has SAN replication between data centers. This is another solution; while low level - may be valid to explore. They use it with Oracle and MSSQLServer to do cross datacenter replication. cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB replication and high-availability across data centers
On Thu, 2010-10-21 at 18:07 +0100, Chris Withers wrote: > On 21/10/2010 16:44, Andreas Jung wrote: > > The well-known options are: > > > > Relstorage on top of some RDBMS-related clustering/replication mechanism > > This is where I'd look, since the RDBMS guys likely have patterns that > match the one you're seeking... True, but they also have SQL which can be replicated with much less overhead. Replicating at such a low level requires replicating whole objects even if a single attribute changes whether you use Relstorage or not. This is why I would consider replication at application level, especially in a limited bandwith environment. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB replication and high-availability across data centers
On 10/21/2010 09:44 AM, Andreas Jung wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Hi there, > > we are currently investigating options for a large-scale solution > where the machines (ZEO clients and ZEO server) should be separated > across two data centers. Incoming traffic should be balanced between > both data centers. Automatic failover and online-replication are > desirable - "high-performance" is a must. > > The well-known options are: > > ZRS > (ZEO) > Relstorage on top of some RDBMS-related clustering/replication mechanism Six Feet Up is running RelStorage + MySQL + async replication for one of their customers. AFAIK, it works well. Also, I expect the new replication features in Postgres 9 to work well with RelStorage. Depending on what you're doing, balancing between data centers could be very complicated. You might start simple by directing traffic to only one data center at a time. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
