[
https://issues.apache.org/jira/browse/KUDU-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120097#comment-18120097
]
ASF subversion and git services commented on KUDU-3734:
-------------------------------------------------------
Commit 6e5cd39fdef425820f9cc48c8b2cd83b6d0cfaa7 in kudu's branch
refs/heads/master from Ashwani Raina
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=6e5cd39fd ]
KUDU-3734 Compaction budget refinements and observability
KUDU-3734 added UNDO delta sizes to the knapsack budget used for rowset
compaction selection to prevent OOM. While effective, including UNDO
deltas at full weight can cause compaction to skip rowsets
unnecessarily, particularly when UNDO deltas are large relative to
base+redo data. Skipped rowsets accumulate, increase average rowset
height, and eventually slow down scans because each scan must open
iterators across more overlapping DiskRowSets.
This change addresses three concerns raised as follow-ups:
1. Budget refinements to reduce unnecessary skipping
a) --rowset_undo_deltas_budget_weight(double, default 1.0, runtime):
A fractional multiplier applied to UNDO delta sizes when computing
the budget weight of a rowset candidate. At the default of 1.0
there is no change in behaviour.
b) --rowset_deltas_size_include_undo is now tagged 'runtime':
Previously this flag required a server restart to toggle. Making
it runtime-changeable enables operators and future auto-disable
mechanisms to flip it live without disruption.
2. Metrics to detect slow scans caused by budget-induced compaction lag
a) rowsets_per_scan (tablet-level, histogram, max=1024, kDebug):
Tracks the distribution of DiskRowSet iterators opened per scan
request, recorded in Tablet::CaptureConsistentIterators(). The MRS
and any txn_memrowset iterators are excluded so the histogram
measures only the on-disk overlap a scan must resolve. A rising
P99 is the most direct observable consequence of insufficient
compaction: more overlapping DiskRowSets means more data sources
that must be merged to produce a single scan result.
3. Detection and operator notification when the system is degraded
a) compaction_budget_skip_health (tablet-level, gauge, uint32, kInfo):
Set to 1 (degraded) when at least one DiskRowSet candidate
individually exceeds --tablet_compaction_budget_mb and is therefore
permanently unselectable under the current settings. Set to 0
(healthy) otherwise.
b) tablets_num_compaction_budget_degraded
(tablet-server-level, gauge, int32, kWarn):
Number of tablet replicas on the tablet server whose
compaction_budget_skip_health gauge is 1 (degraded).
c) Throttled WARNING log (at most once per 300 s):
When oversized rowsets are detected, a WARNING is emitted naming
the exact flags to adjust, so users know what to do from the
tserver log.
Corresponding unit tests are added to:
compaction-test.cc
compaction_policy-test.cc
ts_tablet_manager-test.cc
Conditions (when met) that require an action from operator:
Condition 1 alone is sufficient to act; conditions 2 and 3
indicate urgency and user-visible impact respectively.
1. Permanent skip state is confirmed: This alone is sufficient evidence
to warrant action (not just investigation): one or more rowsets
individually exceed the budget and will never be selected for
compaction under the current settings.
i.e., tablets_num_compaction_budget_degraded > 0
2. Compaction is not keeping up: Average diskrowset height keeps moving
upwards. A sustained upward trend in 'average_diskrowset_height' over
a rolling window of your choosing.
3. Scans are accessing more rowsets: An early warning of user-visible
scan impact while the problem is still confined to tail scans
(wide/full-table queries). Condition 1 identifies the structural root
cause; this identifies the downstream scan impact. A sustained rise
in the 99th percentile of 'rowsets_per_scan' over a rolling window of
your choosing.
Clear condition: the situation is resolved when
'tablets_num_compaction_budget_degraded' returns to 0. This happens
automatically once oversized rowsets are GC'd or the budget/weight is
adjusted, as 'compaction_budget_skip_health' resets to 0 on the next
PickRowSets() run.
Actions to take:
1. Reduce UNDO weight (--rowset_undo_deltas_budget_weight) to 0.5 (or
half its current value if already lowered) and stop if
'compaction_budget_skip_health' clears after the next compaction cycle.
2. If reducing UNDO weight doesn't help, disable UNDO accounting
(--rowset_deltas_size_include_undo) entirely.
Note: this removes the OOM guard that UNDO inclusion was originally
added for. Monitor tserver memory pressure after making this change.
3. If neither of the above actions are desirable in a given environment,
raise the compaction budget (--tablet_compaction_budget_mb) further.
These guidelines address the side-effects that can arise from enabling
--rowset_deltas_size_include_undo. The exact conditions and appropriate
actions may vary depending on the environment, workload characteristics,
and cluster configuration.
Change-Id: Id63db0006c77ad1d55eb2119d99ee64499435819
Reviewed-on: http://gerrit.cloudera.org:8080/24618
Reviewed-by: Alexey Serbin <[email protected]>
Tested-by: Ashwani Raina <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
> Rowset merge compaction does not consider undo delta size while picking
> rowsets
> -------------------------------------------------------------------------------
>
> Key: KUDU-3734
> URL: https://issues.apache.org/jira/browse/KUDU-3734
> Project: Kudu
> Issue Type: Bug
> Reporter: Ashwani Raina
> Assignee: Ashwani Raina
> Priority: Major
> Fix For: 1.19.0
>
> Attachments: KUDU-3734 Test Plan - final.pdf, KUDU-3734 Test Plan.pdf
>
>
> Rowset merge compaction has preliminary stage where it goes through all the
> eligible rowsets in a tablet that fit the criteria of optimal compaction
> w.r.t., budget, rowset width, range overlaps, etc.
> It uses fractional knapsack algorithm to come up with the list of rowsets
> that can deliver maximum bang for the buck. While a rowset merge compaction
> touches almost all the parts of a data pertaining to the rows within a
> tablet, e.g., base data, redo and undo deltas, it doesn't take into
> consideration the size of undo deltas while calculating density of a rowset
> item in knapsack algorithm.
> Due to this, even if a rowset has undo deltas amount to huge size, it is
> possible that it will get picked up for rowset merge compaction. This can
> lead to OOM scenarios if say size of undo deltas is GBs and when compaction
> finally starts reading the rows and corresponding uncompressed data from
> deltas into memory, the process might cross the memory hard limit, thereby
> forcing OS to take action and kill the kudu service. One such example is
> KUDU-3406.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
