Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24618 )
Change subject: KUDU-3734 Compaction budget refinements and observability ...................................................................... Patch Set 9: Code-Review-2 (2 comments) It seems there is +2 from other reviewers, but I don't think it makes sense to merge this patch before a valid, non-noisy metric is provided to track memory budget-based rejections. What's the point of adding noisy and confusing metrics where it's not possible to build any reliable monitoring upon? Maybe I'm missing something, but as of PS9 this patch is a no-go IMO. http://gerrit.cloudera.org:8080/#/c/24618/9/src/kudu/tablet/compaction_policy.cc File src/kudu/tablet/compaction_policy.cc: http://gerrit.cloudera.org:8080/#/c/24618/9/src/kudu/tablet/compaction_policy.cc@690 PS9, Line 690: metrics_->compact_rs_rowsets_budget_skipped->set_value(skipped_count); : metrics_->compact_rs_bytes_budget_skipped->set_value(skipped_bytes); The comment above claims 'skipped_count' and 'skipped_bytes' contains not only rowsets skipped because of budget restrictions, but all other rowsets that aren't selected because of other reasons. Is there a way to provide a metric that tracks only memory budget-related rejections, without the 'regular' noise of rowsets being rejected based on the internal knapsack algo-based metrics? http://gerrit.cloudera.org:8080/#/c/24618/2/src/kudu/tablet/tablet_metrics.cc File src/kudu/tablet/tablet_metrics.cc: http://gerrit.cloudera.org:8080/#/c/24618/2/src/kudu/tablet/tablet_metrics.cc@447 PS2, Line 447: : 60000LU, 1); > Good catch! I don't think that switching from an accumulative metric to a gauge resolve the underlying issue. From what I can see, there is too much noise in this metric because a non-zero value include not only attempts to go over compaction memory budget, but other cases stemming from the internal intra-rowset scoring used by finding the rowset combination that benefits from the compaction the most. So, the same story here: I don't see value in adding such a noisy metric. I don't see how an operator can build a reliable monitoring rule based on this metric. IIUC, depending on the data ingestion/update rate, there can always be rowsets that aren't selected because there are others that have higher internal scores to be compacted, even if rowsets aren't piling up. With that, the gauge can stay non-zero for a very long period of time even if the none of rowset combination would ever go over the budget. >From the other side, the fact that the metric is reset to 0 makes it even more >confusing: computing the scores takes time, so there is a window for this >metric to stay 0 for some period of time, even it's quite short. How a >monitoring system or an operator can tell whether that dip to 0 constitutes a >state of the metric between compactions vs being reset before it's set to a >non-zero value? -- To view, visit http://gerrit.cloudera.org:8080/24618 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id63db0006c77ad1d55eb2119d99ee64499435819 Gerrit-Change-Number: 24618 Gerrit-PatchSet: 9 Gerrit-Owner: Ashwani Raina <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Ashwani Raina <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber <[email protected]> Gerrit-Reviewer: Zoltan Martonka <[email protected]> Gerrit-Comment-Date: Fri, 11 Sep 2026 18:52:03 +0000 Gerrit-HasComments: Yes
