On Tue, Sep 14, 2010 at 9:15 PM, David Shrewsbury <[email protected] > wrote:
> Hi all, > > Some recent changes to the Drizzle kernel have been made so > that we can now support bulk loads in the replication stream > (e.g., the transaction log). As a result, we now have an issue > where we'd like to hear some feedback from the community. > > Before the change, a single Google Protobuf Transaction message > was guaranteed to contain all statements executed within the > transaction. In order to support bulk loads, there is now a limit > on the size we allow a Transaction message to grow to, so a > large transaction could likely create more than one GPB Transaction > message, but each would be linked together using the same > transaction ID. > > Now the issue: There is currently no guarantee that readers of > the replication stream will receive all Transaction messages from > a single transaction in an atomic fashion. In other words, if you > have a large transaction (like a bulk load) happening on one > connection, and other transactions occurring in different connections, > the replication stream will contain intermingled GPB Transaction > messages. > > The options for handling this are: > > 1) Place locks on the replication stream and buffer Transaction > messages, likely to disk, until the transaction is complete and > send them all down the stream together, locking the stream to > do so. This is how MySQL handles writing RBR events to the > binlog. > > 2) Let the replication stream readers do the buffering of the > Transaction messages themselves. > > #1 causes concurrency issues on the master. MySQL suffers > from this same problem, and they note it in the documentation > as a disadvantage of using the RBR format: > > http://dev.mysql.com/doc/refman/5.1/en/replication-sbr-rbr.html > > #2 does not suffer from the concurrency issue of #1, but it does > mean marginally more work for TransactionApplier implementers > to buffer GBP messages until the end Transaction message is > read. > Why not support both strategies using different replicator plugins for example? I'm guessing it is a matter of where you want to do the extra work of buffering, the actual logic would be the same if it is done on the slave or the master, right? And maybe rename the GPB message if it no longer maps directly to a database transaction? /Marcus
_______________________________________________ Mailing list: https://launchpad.net/~drizzle-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
