Hello Marton Greber, Alexey Serbin, Zoltan Martonka, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24618
to look at the new patch set (#10).
Change subject: KUDU-3734 Compaction budget refinements and observability
......................................................................
KUDU-3734 Compaction budget refinements and observability
KUDU-3734 added UNDO delta sizes to the knapsack budget used for rowset
compaction selection to prevent OOM. While effective, including UNDO
deltas at full weight can cause compaction to skip rowsets
unnecessarily, particularly when UNDO deltas are large relative to
base+redo data. Skipped rowsets accumulate, increase average rowset
height, and eventually slow down scans because each scan must open
iterators across more overlapping DiskRowSets.
This change addresses three concerns raised as follow-ups:
1. Budget refinements to reduce unnecessary skipping
a) --rowset_undo_deltas_budget_weight(double, default 1.0, runtime):
A fractional multiplier applied to UNDO delta sizes when computing
the budget weight of a rowset candidate. At the default of 1.0
there is no change in behaviour.
b) --rowset_deltas_size_include_undo is now tagged 'runtime':
Previously this flag required a server restart to toggle. Making
it runtime-changeable enables operators and future auto-disable
mechanisms to flip it live without disruption.
2. Metrics to detect slow scans caused by budget-induced compaction lag
a) rowsets_per_scan (tablet-level, histogram, max=1024, kDebug):
Tracks the distribution of DiskRowSet iterators opened per scan
request, recorded in Tablet::CaptureConsistentIterators(). The MRS
and any txn_memrowset iterators are excluded so the histogram
measures only the on-disk overlap a scan must resolve. A rising
P99 is the most direct observable consequence of insufficient
compaction: more overlapping DiskRowSets means more data sources
that must be merged to produce a single scan result.
3. Detection and operator notification when the system is degraded
a) compaction_budget_skip_health (tablet-level, gauge, uint32, kInfo):
Set to 1 (degraded) when at least one DiskRowSet candidate
individually exceeds --tablet_compaction_budget_mb and is therefore
permanently unselectable under the current settings. Set to 0
(healthy) otherwise.
b) tablets_num_compaction_budget_degraded
(tablet-server-level, gauge, int32, kWarn):
Number of tablet replicas on the tablet server whose
compaction_budget_skip_health gauge is 1 (degraded).
c) Throttled WARNING log (at most once per 300 s):
When oversized rowsets are detected, a WARNING is emitted naming
the exact flags to adjust, so users know what to do from the
tserver log.
Conditions (when met) that require an action from operator:
Condition 1 alone is sufficient to act; conditions 2 and 3
indicate urgency and user-visible impact respectively.
1. Permanent skip state is confirmed: This alone is sufficient evidence
to warrant action (not just investigation): one or more rowsets
individually exceed the budget and will never be selected for
compaction under the current settings.
i.e., tablets_num_compaction_budget_degraded > 0
2. Compaction is not keeping up: Average diskrowset height keeps moving
upwards. A sustained upward trend in 'average_diskrowset_height' over
a rolling window of your choosing.
3. Scans are accessing more rowsets: An early warning of user-visible
scan impact while the problem is still confined to tail scans
(wide/full-table queries). Condition 1 identifies the structural root
cause; this identifies the downstream scan impact. A sustained rise
in the 99th percentile of 'rowsets_per_scan' over a rolling window of
your choosing.
Clear condition: the situation is resolved when
'tablets_num_compaction_budget_degraded' returns to 0. This happens
automatically once oversized rowsets are GC'd or the budget/weight is
adjusted, as 'compaction_budget_skip_health' resets to 0 on the next
PickRowSets() run.
Actions to take:
1. Reduce UNDO weight (--rowset_undo_deltas_budget_weight) to 0.5 (or
half its current value if already lowered) and stop if
'compaction_budget_skip_health' clears after the next compaction cycle.
2. If reducing UNDO weight doesn't help, disable UNDO accounting
(--rowset_deltas_size_include_undo) entirely.
Note: this removes the OOM guard that UNDO inclusion was originally
added for. Monitor tserver memory pressure after making this change.
3. If neither of the above actions are desirable in a given environment,
raise the compaction budget (--tablet_compaction_budget_mb) further.
These guidelines address the side-effects that can arise from enabling
--rowset_deltas_size_include_undo. The exact conditions and appropriate
actions may vary depending on the environment, workload characteristics,
and cluster configuration.
Change-Id: Id63db0006c77ad1d55eb2119d99ee64499435819
---
M src/kudu/tablet/compaction_policy.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/diskrowset.h
M src/kudu/tablet/rowset.cc
M src/kudu/tablet/rowset.h
M src/kudu/tablet/rowset_info.cc
M src/kudu/tablet/rowset_info.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet_metrics.cc
M src/kudu/tablet/tablet_metrics.h
M src/kudu/tserver/ts_tablet_manager.cc
11 files changed, 228 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/18/24618/10
--
To view, visit http://gerrit.cloudera.org:8080/24618
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id63db0006c77ad1d55eb2119d99ee64499435819
Gerrit-Change-Number: 24618
Gerrit-PatchSet: 10
Gerrit-Owner: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
Gerrit-Reviewer: Zoltan Martonka <[email protected]>
