Hi all, Some recent changes to the Drizzle kernel have been made so that we can now support bulk loads in the replication stream (e.g., the transaction log). As a result, we now have an issue where we'd like to hear some feedback from the community.
Before the change, a single Google Protobuf Transaction message was guaranteed to contain all statements executed within the transaction. In order to support bulk loads, there is now a limit on the size we allow a Transaction message to grow to, so a large transaction could likely create more than one GPB Transaction message, but each would be linked together using the same transaction ID. Now the issue: There is currently no guarantee that readers of the replication stream will receive all Transaction messages from a single transaction in an atomic fashion. In other words, if you have a large transaction (like a bulk load) happening on one connection, and other transactions occurring in different connections, the replication stream will contain intermingled GPB Transaction messages. The options for handling this are: 1) Place locks on the replication stream and buffer Transaction messages, likely to disk, until the transaction is complete and send them all down the stream together, locking the stream to do so. This is how MySQL handles writing RBR events to the binlog. 2) Let the replication stream readers do the buffering of the Transaction messages themselves. #1 causes concurrency issues on the master. MySQL suffers from this same problem, and they note it in the documentation as a disadvantage of using the RBR format: http://dev.mysql.com/doc/refman/5.1/en/replication-sbr-rbr.html #2 does not suffer from the concurrency issue of #1, but it does mean marginally more work for TransactionApplier implementers to buffer GBP messages until the end Transaction message is read. Thoughts? -Dave _______________________________________________ Mailing list: https://launchpad.net/~drizzle-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
