Hello Zoltan Martonka, Kurt Deschler, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22406
to look at the new patch set (#5).
Change subject: [compaction] Major Delta Compaction batching to avoid OOM
......................................................................
[compaction] Major Delta Compaction batching to avoid OOM
The patch adds logic to do major delta compaction in batches
instead of processing all the delta blocks in one go. The batch
is defined by one or more row updates depending on how much space
those deltas consume in terms of blocks, mutations, maintaining
UNDO lists. For each batch, all the corresponding REDO deltas are
read and corresponding UNDO mutations are created to applied.
After all the delta blocks and their corresponding deltas are applied,
base data is committed.
Main highlights of this change are listed as:
1. Maintains last block pointer to the delta block that was read
in the last batch.
2. Maintains the next row id to start from when next batch processing
starts.
3. Keeps major delta compaction object's mode i.e. whether it is running
in batch mode or normal mode (where no batching is done).
4. Maintains information whether all the delta blocks have been already
read and processed. This denotes exit condition when running in batch
mode.
Note: There are a couple of unit tests that I need to refine a little bit
before pushing for review.
Change-Id: I111d613ec717534df9ba09e5660420b5d6c9fed0
---
M src/kudu/tablet/delta_compaction.cc
M src/kudu/tablet/delta_compaction.h
M src/kudu/tablet/delta_iterator_merger.cc
M src/kudu/tablet/delta_iterator_merger.h
M src/kudu/tablet/delta_store.h
M src/kudu/tablet/deltafile.cc
M src/kudu/tablet/deltafile.h
M src/kudu/tablet/deltamemstore.h
M src/kudu/tablet/diskrowset.cc
9 files changed, 452 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/06/22406/5
--
To view, visit http://gerrit.cloudera.org:8080/22406
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I111d613ec717534df9ba09e5660420b5d6c9fed0
Gerrit-Change-Number: 22406
Gerrit-PatchSet: 5
Gerrit-Owner: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Zoltan Martonka <[email protected]>
