Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/4363 to look at the new patch set (#18). Change subject: KUDU-1601. Delete ancient UNDO delta blocks in the background ...................................................................... KUDU-1601. Delete ancient UNDO delta blocks in the background This patch adds a maintenance manager background task that deletes "ancient" UNDO delta blocks, which means blocks that correspond to data that is considered no longer reachable and a candidate for garbage collection. The task only deletes entire blocks and so does not provoke write amplification. This maintenance task operates in the following way: 1. UpdateStats() for the op returns the maximum potentially gc'able bytes of undos in the rowset, which is the sum of all undo delta store sizes up until an initialized one with max_timestamp > the AHM (ancient history mark). 2. Perform() for the op initializes deltas for some budgeted amount of time on the tablet. Per rowset it performs a binary search and initializes undo delta stores until it finds the earliest one with max_timestamp > AHM. That makes the next UpdateStats() call more accurate. Then it starts from the oldest and initializes undos within the budget. Finally, once it's through with its time budget, it garbage-collects as many undos in the rowset as possible. To avoid starvation of performance ops, there is a flag that incorporates some randomness into the scheduler, at the MM level. This limits the fraction of the time we consider data GC ops higher priority than performance ops. This patch includes the following: * New UNDO delta block GC MM task * New UNDO delta block GC metrics (at the tablet level only) * Flags to enable / disable the GC task as well as flags to throttle it * A few minor improvements in the maintenance manager * Fixes for a few preexisting clang-tidy lint complaints Notable implementation details: * When performing undo delta GC in Tablet::DeleteAncientUndoDeltas(), we only flush the tablet metadata after making the metadata changes across all rowsets. This should be safe because we are not actually modifying any data, we are simply removing references to blocks that are no longer reachable by new scanners. Includes the following tests: * RowSet-level unit test in diskrowset-test * Tablet-level functional test in tablet_history_gc-test * Tablet-level concurrency test in mt-tablet-test * Integration test utilizing the tserver-level MM task in tablet_history_gc-itest * Incorporated into RandomizedTabletHistoryGcITest in tablet_history_gc-itest Change-Id: I0309bf7acfb6d018860c80f354012c3500da5c68 --- M src/kudu/integration-tests/tablet_history_gc-itest.cc M src/kudu/master/master.cc M src/kudu/tablet/delta_tracker.cc M src/kudu/tablet/delta_tracker.h M src/kudu/tablet/deltafile.cc M src/kudu/tablet/deltafile.h M src/kudu/tablet/diskrowset-test.cc M src/kudu/tablet/diskrowset.cc M src/kudu/tablet/diskrowset.h M src/kudu/tablet/memrowset.h M src/kudu/tablet/mock-rowsets.h M src/kudu/tablet/mt-tablet-test.cc M src/kudu/tablet/rowset.h M src/kudu/tablet/rowset_metadata.cc M src/kudu/tablet/rowset_metadata.h M src/kudu/tablet/tablet.cc M src/kudu/tablet/tablet.h M src/kudu/tablet/tablet_history_gc-test.cc M src/kudu/tablet/tablet_metrics.cc M src/kudu/tablet/tablet_metrics.h M src/kudu/tablet/tablet_mm_ops.cc M src/kudu/tablet/tablet_mm_ops.h M src/kudu/tserver/tablet_server.cc M src/kudu/util/maintenance_manager-test.cc M src/kudu/util/maintenance_manager.cc M src/kudu/util/maintenance_manager.h 26 files changed, 1,322 insertions(+), 140 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/4363/18 -- To view, visit http://gerrit.cloudera.org:8080/4363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0309bf7acfb6d018860c80f354012c3500da5c68 Gerrit-PatchSet: 18 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Mike Percy <mpe...@apache.org> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: David Ribeiro Alves <dral...@apache.org> Gerrit-Reviewer: Jean-Daniel Cryans <jdcry...@apache.org> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy <mpe...@apache.org> Gerrit-Reviewer: Tidy Bot