Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Tino Wildenhain wrote: the standard approach for master-master replicated data is to use an UUID-Datatype (see http://en.wikipedia.org/wiki/UUID ) Good point, although we only have space for 64 bits, so the chance of collision would be higher than it usually is in a UUID system. That is a serious issue that will lead to database corruption. The commit and pack locks need to be cluster-wide. Does MySQL have a way to do that? Well, synchronous replication is a hard beast, master-master synchronous even harder (and you have to ask what problem you really want to solve with it, since the trade-offs are massive) True. Master-master replication can be done, but it often turns out to perform *worse* than a single master, and the mere notion of a cluster-wide lock frightens little children who want their daddy to come home rather than debug clusters all night. I'd rather look at master-slave replication with a little enhancement: RelStorage already has distinct load and store connections, so it could choose a slave when loading and the master when storing. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Russ Ferriday wrote: (You could get away with a byte for the server ID, leaving masses of bits for the item ID.) That's probably a good idea, but I'd prefer to use the least significant byte for the server ID, effectively allocating OIDs modulo 256. :-) Also, it's becoming clear that each server should have an independent new_oid table (rather than replicate it), and we should do something to ensure the server ID component of the OID is always correct. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Hi, Shane Hathaway wrote: Stefan H. Holek wrote: We have put up two boxes in a MySQL master-master replication setup [1]. As long as we only write to one of the masters all is fine. Writing to both masters (and expecting MySQL to sort it out) is giving us trouble. We see things like: Last_Errno: 1062 Last_Error: Error 'Duplicate entry '41860' for key 1' on query. Default database: 'prod_zodb'. Query: 'INSERT INTO new_oid VALUES ()' Perhaps we can solve this by making each server allocate a different set of OIDs. For example, one server might allocate odd-numbered OIDs while the other allocates even-numbered OIDs. Even better would be some kind of standard MySQL solution, but I haven't found any yet. the standard approach for master-master replicated data is to use an UUID-Datatype (see http://en.wikipedia.org/wiki/UUID ) Last_Errno: 1050 Last_Error: Error 'Table 'temp_store' already exists' on query. Default database: 'prod_zodb'. Query: 'CREATE TEMPORARY TABLE temp_store ( zoidBIGINT NOT NULL PRIMARY KEY, prev_tidBIGINT NOT NULL, md5 CHAR(32), state LONGBLOB ) ENGINE MyISAM' Uh-oh. The commit lock should have prevented this, so this error suggests that each master has a completely independent set of locks! surprise ;) That is a serious issue that will lead to database corruption. The commit and pack locks need to be cluster-wide. Does MySQL have a way to do that? Well, synchronous replication is a hard beast, master-master synchronous even harder (and you have to ask what problem you really want to solve with it, since the trade-offs are massive) Tino smime.p7s Description: S/MIME Cryptographic Signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Stefan! This is like holding all your eyelids apart and shouting out for somebody with a sharp stick! ;) Watcha gonna do when you add a third server? er.. lemme see... Multiply all OIDs by three and allocate new OIDs modulo three? Eeek! Why not make the OID a composite of server ID and sequence? Even if these were both packed in the same 64 bits, just so you keep it a *little bit nasty*, and then maybe just get one eye poked out! ;) (You could get away with a byte for the server ID, leaving masses of bits for the item ID.) Best wishes! --r On 22 May 2008, at 14:49, Stefan H. Holek wrote: Hi Russ, This is correct. I should have mentioned that the two servers are configured so that server A creates odd, server B even sequence numbers. This makes the duplicate id error pretty mysterious, IMO ;-) Stefan On 22.05.2008, at 13:15, Russ Ferriday wrote: I have not read the reference, but it seems that this scheme can never work without keys (OIDs) being qualified by which master they originated on. Otherwise one of the mySQLs would need to be responsible for handing out unique oids. You need an OID origination namespace for each master. -- Stefan H. Holek [EMAIL PROTECTED] Russ Ferriday - Topia Systems - Open Source content management with Plone and Zope [EMAIL PROTECTED] - office: +44 2076 1777588 - mobile: +44 7789 338868 - skype: ferriday ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Stefan H. Holek wrote: We have put up two boxes in a MySQL master-master replication setup [1]. As long as we only write to one of the masters all is fine. Writing to both masters (and expecting MySQL to sort it out) is giving us trouble. We see things like: Last_Errno: 1062 Last_Error: Error 'Duplicate entry '41860' for key 1' on query. Default database: 'prod_zodb'. Query: 'INSERT INTO new_oid VALUES ()' Perhaps we can solve this by making each server allocate a different set of OIDs. For example, one server might allocate odd-numbered OIDs while the other allocates even-numbered OIDs. Even better would be some kind of standard MySQL solution, but I haven't found any yet. Last_Errno: 1050 Last_Error: Error 'Table 'temp_store' already exists' on query. Default database: 'prod_zodb'. Query: 'CREATE TEMPORARY TABLE temp_store ( zoidBIGINT NOT NULL PRIMARY KEY, prev_tidBIGINT NOT NULL, md5 CHAR(32), state LONGBLOB ) ENGINE MyISAM' Uh-oh. The commit lock should have prevented this, so this error suggests that each master has a completely independent set of locks! That is a serious issue that will lead to database corruption. The commit and pack locks need to be cluster-wide. Does MySQL have a way to do that? Thanks for working on this! Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Hi Russ, This is correct. I should have mentioned that the two servers are configured so that server A creates odd, server B even sequence numbers. This makes the duplicate id error pretty mysterious, IMO ;-) Stefan On 22.05.2008, at 13:15, Russ Ferriday wrote: I have not read the reference, but it seems that this scheme can never work without keys (OIDs) being qualified by which master they originated on. Otherwise one of the mySQLs would need to be responsible for handing out unique oids. You need an OID origination namespace for each master. -- Stefan H. Holek [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Stefan, Thanks for opening this thread here. I'm not actually doing anything like this, but it's great to see you pushing the envelop. I have done a lot of work on replicated data, over the years, that's where the following comment comes from. On 22 May 2008, at 10:36, Stefan H. Holek wrote: We have put up two boxes in a MySQL master-master replication setup [1]. As long as we only write to one of the masters all is fine. Writing to both masters (and expecting MySQL to sort it out) is giving us trouble. We see things like: Last_Errno: 1062 Last_Error: Error 'Duplicate entry '41860' for key 1' on query. Default database: 'prod_zodb'. Query: 'INSERT INTO new_oid VALUES ()' I have not read the reference, but it seems that this scheme can never work without keys (OIDs) being qualified by which master they originated on. Otherwise one of the mySQLs would need to be responsible for handing out unique oids. You need an OID origination namespace for each master. On the other issue, no idea. Best, --r. and Last_Errno: 1050 Last_Error: Error 'Table 'temp_store' already exists' on query. Default database: 'prod_zodb'. Query: 'CREATE TEMPORARY TABLE temp_store ( zoidBIGINT NOT NULL PRIMARY KEY, prev_tidBIGINT NOT NULL, md5 CHAR(32), state LONGBLOB ) ENGINE MyISAM' Stefan [1] http://www.howtoforge.com/mysql_master_master_replication On 20.05.2008, at 18:36, Shane Hathaway wrote: That is mostly correct. You need to replicate transaction, object_state, current_object, and do something with new_oid. There is no requirement to replicate the other tables, since they are used only for packing, although replicating object_ref and object_refs_added may save some work. The new_oid table is special to the MySQL adapter. RelStorage uses the new_oid table as a kind of sequence. Like any sequence, it is intentionally non-transactional. Perhaps there would be no ill effects in switching it to the InnoDB engine. Which MySQL replication method are you looking into? -- Stefan H. Holek [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev Russ Ferriday - Topia Systems - Open Source content management with Plone and Zope [EMAIL PROTECTED] - office: +44 2076 1777588 - mobile: +44 7789 338868 - skype: ferriday ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Which tables to replicate?
We have put up two boxes in a MySQL master-master replication setup [1]. As long as we only write to one of the masters all is fine. Writing to both masters (and expecting MySQL to sort it out) is giving us trouble. We see things like: Last_Errno: 1062 Last_Error: Error 'Duplicate entry '41860' for key 1' on query. Default database: 'prod_zodb'. Query: 'INSERT INTO new_oid VALUES ()' and Last_Errno: 1050 Last_Error: Error 'Table 'temp_store' already exists' on query. Default database: 'prod_zodb'. Query: 'CREATE TEMPORARY TABLE temp_store ( zoidBIGINT NOT NULL PRIMARY KEY, prev_tidBIGINT NOT NULL, md5 CHAR(32), state LONGBLOB ) ENGINE MyISAM' Stefan [1] http://www.howtoforge.com/mysql_master_master_replication On 20.05.2008, at 18:36, Shane Hathaway wrote: That is mostly correct. You need to replicate transaction, object_state, current_object, and do something with new_oid. There is no requirement to replicate the other tables, since they are used only for packing, although replicating object_ref and object_refs_added may save some work. The new_oid table is special to the MySQL adapter. RelStorage uses the new_oid table as a kind of sequence. Like any sequence, it is intentionally non-transactional. Perhaps there would be no ill effects in switching it to the InnoDB engine. Which MySQL replication method are you looking into? -- Stefan H. Holek [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Shared/DC/ZRDB/TM.py:_register
(Re-send because I forgot to CC the list) Le Monday 19 May 2008 21:13:49, vous avez écrit : > Think a bit about your wish: > > As soon as one transaction ends, a new one starts. > > What should happen with your artificial error handling transaction? > Should it be aborted? or committed?. I think it should be aborted. Partly because it's what happens currently, and because I don't think there is any need to permanently modify data in the error hook. If one wants to do so, I guess he will just write an error path in his code. > What should happen when the > commit fails -- another error handling, in another error handling > transaction? I see this as an argument against commiting the error-hook transaction. > The current behaviour is good in most cases. > If you dislike it in some special cases, abort the transaction > (you will get a new one, aborted automatically at the end > of error handling, unless you do the commit). The point is that I don't see why it would be bad in those "most cases" to systematicaly abort. I did the modification I suggested in the bug report before submiting it, and the error page still renders the error traceback properly. Could you be more specific about what's missing from error hook if original transaction is aborted ? > The better alternative would be to not prevent "join"s to > a doomed transaction. Right. Only commits should be prevented (they currently are when exception happened during TPC). I'm not sure about the case where exception was raised before TPC: should commits be prevented in this case too ? Maybe it's the coder's responsibility not to call commit in the error hook. -- Vincent Pelletier ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Shared/DC/ZRDB/TM.py:_register
Le Thursday 22 May 2008 10:40:46, vous avez écrit : > Maybe, you search the "zope-dev" archive to find the thread that convinced > us to change the former behaviour (the one you want now again) > into the current one? Adding links for possible future reference: https://bugs.launchpad.net/zope2/+bug/142446 First discusses a problem with handling erorr outside of any transaction, suggesting a fix similar to mine, and then later it was agreed that failed transaction should be aborted after error handling (https://bugs.launchpad.net/zope2/+bug/142446/comments/3). > The same can apply to "__traceback_info__" and "__traceback_supplement__" > information derived from persistent objects. > This information will then reflect the persistent state as it > has been when the transaction started and not as it was when the > exception occurred. I see. Thanks. And the problem with providing a "pointer" to such information is the same as having 2 transactions simultaneously started for the same thread, I guess. > The ZODB has a notion of "doomed transaction". > A transaction gets doomed when something happens that can > can lead to persistent inconsistencies should the > transaction be committed. > > A failing "commit" and an unsuccessful reset to a savepoint > are example "doom" reasons. I am not sure whether a "ReadConflictError", > too, dooms the transaction. BTW, the usual error hook treats conflict error exceptions differently from others, and I guess it was done so because those can happen in TPC. In my case, the problem was triggered by MultipleUndoErrors, which is not specialy handled. I have the feeling that current exception class inheritance scheme is not so good, and that it would be needed to create more abstract exceptions classes to inherit from. For example, an exception class which would instruct publisher to retry transaction and another to instruct it to give up (...and of course abort transaction) as fast as possible. I have no idea about the needed amount of work. -- Vincent Pelletier ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
