Hello Alexey Serbin, Kudu Jenkins, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16492 to look at the new patch set (#10). Change subject: KUDU-2612 p11: persist txn metadata in superblock ...................................................................... KUDU-2612 p11: persist txn metadata in superblock Currently, transaction metadata is entirely stored in the WALs; once a transaction is begun, it anchors WALs indefinitely. This is untenable, as any tablet that participates in a transaction would never be able to GC its WALs. This patch addresses this by storing transaction metadata in the tablet superblock for the following participant ops: - BEGIN_TXN: an empty record is added to the superblock for the given transaction ID. In the future, this can be used to store the owner of the transaction. - FINALIZE_COMMIT: the commit timestamp is added to the superblock for the given transaction ID. The commit timestamp of a transaction will be used when scanning over mutations applied to a tablet instead of the apply timestamp. - ABORT_TXN: an abort bit is added to the superblock for the given transaction ID. Upon applying each of these ops, the op's OpId is anchored in the WALs until the tablet metadata is successfully flushed, ensuring that if we don't flush the tablet metadata before restarting, we can rebuild any ops that whose mutations had not been persisted by replaying the WALs. What about BEGIN_COMMIT ops? While these ops will still need to be anchored, BEGIN_COMMIT ops don't need to persist any long-term information. As such, their anchoring is slightly different -- rather than mutating the tablet metadata and unanchoring on a metadata flush, BEGIN_COMMIT ops will update the in-memory state for the in-flight Txn, and unanchor once the corresponding FINALIZE_COMMIT or ABORT_TXN begins applying. On bootstrap, the following rules are applied: - If we've persisted terminal information (i.e. abort or commit) about a transaction ID, we can skip replaying any participant op for that transaction. - If we otherwise have a persisted record for the transaction ID, we should start an open transaction and replay all participant ops for that transaction. - If we have no record of a transaction persisted, we must replay all participant ops for the given transaction. While storing things in the superblock is not ideal, there are further improvements that can be made to reduce its size, e.g. after finalizing a transaction, the transactions mutations may be merged in with the rest of the tablet; once all mutations of a given transaction have been merged, the transaction metadata may be removed from the metadata. Change-Id: I2f32808aef10484f4e0ad3942bb005f61fbdb34a --- M src/kudu/integration-tests/txn_participant-itest.cc M src/kudu/tablet/metadata.proto M src/kudu/tablet/ops/participant_op.cc M src/kudu/tablet/ops/participant_op.h M src/kudu/tablet/tablet.cc M src/kudu/tablet/tablet.h M src/kudu/tablet/tablet_bootstrap.cc M src/kudu/tablet/tablet_metadata-test.cc M src/kudu/tablet/tablet_metadata.cc M src/kudu/tablet/tablet_metadata.h M src/kudu/tablet/txn_participant-test.cc M src/kudu/tablet/txn_participant.cc M src/kudu/tablet/txn_participant.h 13 files changed, 728 insertions(+), 131 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/92/16492/10 -- To view, visit http://gerrit.cloudera.org:8080/16492 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2f32808aef10484f4e0ad3942bb005f61fbdb34a Gerrit-Change-Number: 16492 Gerrit-PatchSet: 10 Gerrit-Owner: Andrew Wong <aw...@cloudera.com> Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com> Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com> Gerrit-Reviewer: Kudu Jenkins (120)