Hello David Ribeiro Alves, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/4363
to look at the new patch set (#20).
Change subject: KUDU-1601. Delete ancient UNDO delta blocks in the background
......................................................................
KUDU-1601. Delete ancient UNDO delta blocks in the background
This patch adds a maintenance manager background task that deletes
"ancient" UNDO delta blocks, which means blocks that correspond to data
that is considered no longer reachable and a candidate for garbage
collection. The task only deletes entire blocks and so does not provoke
write amplification.
This maintenance task operates in the following way:
1. UpdateStats() returns the maximum potentially gc'able bytes of undos
in the rowset, which is the sum of all undo delta store sizes up
until an initialized one with max_timestamp > the AHM (ancient
history mark). The accuracy of this estimate improves over time, and
in a steady state will be exact, as undo delta blocks are initialized
while running Perform().
2. Perform() initializes undo delta stores for the tablet for some
budgeted amount of time. Per rowset it initializes undo delta stores
with its budget until it finds the earliest one with max_timestamp >
AHM. That makes the next UpdateStats() call more accurate. Once it
has exhausted its time budget, or has initialized all ancient undo
blocks, it garbage-collects all of the known ancient undo delta
blocks in the tablet.
To avoid starvation of performance improvement maintenance ops, a new
flag named --data_gc_prioritization_prob has been introduced that
incorporates some randomness into the scheduler at the maintenance
manager level. This controls the fraction of the time that the scheduler
considers data GC ops higher priority than performance improvement ops.
This patch includes the following:
* New UNDO delta block GC MM task
* New UNDO delta block GC metrics (at the tablet level only)
* Flags to enable / disable the GC task as well as flags to throttle it
* A few minor improvements in the maintenance manager
* Fixes for a few preexisting clang-tidy lint complaints
Notable implementation details:
* When performing undo delta GC in Tablet::DeleteAncientUndoDeltas(), we
only flush the tablet metadata after making the metadata changes
across all rowsets. This is safe because we are not actually modifying
any data, we are simply removing references to blocks that are no
longer reachable by new scanners. The code path that handles the
metadata update for compactions and ancient history data GC,
DeltaTracker::CommitDeltaStoreMetadataUpdate(), has a DCHECK in place
to ensure that it is never called without specifying blocks to remove.
This guarantees that the DeltaMemStore flush code path located in
DeltaTracker::FlushDMS(), the only delta-related code path that
modifies user-visible data, does not utilize that routine for its
flush. This fact was also verified by inspection -- FlushDMS()
contains its own flush code path.
Includes the following tests:
* RowSet-level unit test in diskrowset-test
* Tablet-level functional test in tablet_history_gc-test
* Tablet-level concurrency test in mt-tablet-test
* Integration test utilizing the tserver-level MM task in
tablet_history_gc-itest
* Incorporated into RandomizedTabletHistoryGcITest in
tablet_history_gc-itest
Manual testing:
* I ran 300 iterations of TabletHistoryGcITest.TestUndoDeltaBlockGc on
the dist-test cluster under TSAN with 12 stress threads:
http://dist-test.cloudera.org/job?job_id=mpercy.1487901212.3733
* I also ran YCSB on a 10-node cluster on a table with 200 tablets with
mostly default parameters except for --tablet_history_max_age_sec=60.
YCSB was configured like so:
recordcount=100000
operationcount=6000000
updateproportion=1.0
requestdistribution=zipfian
threadcount=10
kudu_pre_split_num_tablets=200
kudu_sync_ops=true
This workload took 839 seconds to run and I did not observe an average
update latency increase over time (there was a mild sawtooth pattern),
which indicated to me that the compaction operations were keeping up
with the updates. The undo delta GC operations were also keeping pace
and garbage was being collected aggressively, with generally only tens
of MB, or less, of reclaimable data per tablet being present at any
given time. It seems the current defaults are reasonable, although
additional performance testing is likely warranted.
Change-Id: I0309bf7acfb6d018860c80f354012c3500da5c68
---
M src/kudu/integration-tests/tablet_history_gc-itest.cc
M src/kudu/master/master.cc
M src/kudu/tablet/delta_tracker.cc
M src/kudu/tablet/delta_tracker.h
M src/kudu/tablet/deltafile.cc
M src/kudu/tablet/deltafile.h
M src/kudu/tablet/diskrowset-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/diskrowset.h
M src/kudu/tablet/memrowset.h
M src/kudu/tablet/mock-rowsets.h
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/rowset.h
M src/kudu/tablet/rowset_metadata.cc
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_history_gc-test.cc
M src/kudu/tablet/tablet_metrics.cc
M src/kudu/tablet/tablet_metrics.h
M src/kudu/tablet/tablet_mm_ops.cc
M src/kudu/tablet/tablet_mm_ops.h
M src/kudu/tserver/tablet_server.cc
M src/kudu/util/maintenance_manager-test.cc
M src/kudu/util/maintenance_manager.cc
M src/kudu/util/maintenance_manager.h
26 files changed, 1,352 insertions(+), 143 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/4363/20
--
To view, visit http://gerrit.cloudera.org:8080/4363
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0309bf7acfb6d018860c80f354012c3500da5c68
Gerrit-PatchSet: 20
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Mike Percy <mpe...@apache.org>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dral...@apache.org>
Gerrit-Reviewer: Jean-Daniel Cryans <jdcry...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mpe...@apache.org>
Gerrit-Reviewer: Tidy Bot
