[
https://issues.apache.org/jira/browse/KUDU-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101820#comment-18101820
]
ASF subversion and git services commented on KUDU-3429:
-------------------------------------------------------
Commit c4500b278067a97d3feb35fe6c77d5ef322b1f6d in kudu's branch
refs/heads/master from Ashwani
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=c4500b278 ]
KUDU-3429 Add extensive test for delta generation (3/n)
The patch adds a unit test that is a better version of existing tests
to generate UNDO deltas using:
* Pattern of inserting interleaved rows to ensure more rowsets become
eligible for compaction.
* Wide column data to make each mutation size big enough (64KB). Along
with repetitive updates ensures increased delta memory footprint.
* Increased rows per block batch size that ensures compaction op has
more memory footprint, used to hold rowblock and their mutations.
* Use periodic byte pattern to minimize the effect of compression on
wall time and achieve 'compressible on disk but expands in memory'
goal at the same time, for the data generated for UNDO deltas.
* This can act as a de facto unit test for testing high memory
compaction ops. This can be used as a reference base test for any high
memory based compaction tests.
This changelist doesn't contain any functional modifications.
CLEAN_THIRDPARTY
Change-Id: I8c337dba87a9d1cf549f9b53813da8d50e86b6b4
Reviewed-on: http://gerrit.cloudera.org:8080/24412
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Alexey Serbin <[email protected]>
> Refactor CompactRowSetsOp to run on a pre-determined memory budget
> -------------------------------------------------------------------
>
> Key: KUDU-3429
> URL: https://issues.apache.org/jira/browse/KUDU-3429
> Project: Kudu
> Issue Type: Improvement
> Reporter: Alexey Serbin
> Assignee: Ashwani Raina
> Priority: Major
>
> [KUDU-3406|https://issues.apache.org/jira/browse/KUDU-3406] added memory
> budgeting for running CompactRowSetsOp maintenance operations. On its
> nature, that provides an interim approach adding memory budgeting on top of
> the current CompactRowSetsOp implementation as-is.
> Ideally, the implementation of CompactRowSetsOp should be refactored to merge
> the deltas in participating rowsets sequentially, chunk by chunk, persisting
> the results and allocating memory just for small bunch of processed deltas,
> not loading all the deltas at once.
> This JIRA item is to track the work in the context outlined above.
> Key points to address in this scope:
> * even if it's a merge-like operation by its nature, the current
> implementation of CompactRowSetsOp allocates all the memory necessary to load
> the UNDO deltas at once, and it keeps all the preliminary results in the
> memory as well before persisting the result data to disk
> * the current implementation of CompactRowSetsOp loads all the UNDO deltas
> from the rowsets selected for compaction regardless whether they are ancient
> or not; it discards of the data sourced from the ancient deltas in the very
> end before persisting the result data
> Also, while keeping memory usage on a predetermined budget, the new
> implementation for CompactRowSetsOp should strive to avoid IO multiplication
> as much as possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
