FrankChen021 commented on code in PR #20708:
URL: https://github.com/apache/druid/pull/20708#discussion_r4370856777
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/CostBasedAutoScaler.java:
##########
@@ -476,14 +493,29 @@ static double extractPollIdleRatio(Map<String,
Map<String, Object>> taskStats)
return -1.;
}
+ // Use running average metric to smoothen out spikes
+ return extractAutoscalerMetric(
+ taskStats,
+ SeekableStreamIndexTaskRunner.POLL_IDLE_RATIO_RUNNING_AVG
+ );
+ }
+
+ /**
+ * Gets the value of the given metric averaged across all tasks that have
reported the metric.
+ */
+ private static double extractAutoscalerMetric(
+ Map<String, Map<String, Object>> taskStats,
+ String metricName
+ )
+ {
double sum = 0;
int count = 0;
for (Map<String, Object> groupMetrics : taskStats.values()) {
for (Object taskMetric : groupMetrics.values()) {
if (taskMetric instanceof Map) {
Object autoScalerMetricsMap = ((Map<?, ?>)
taskMetric).get(SeekableStreamIndexTaskRunner.AUTOSCALER_METRICS_KEY);
if (autoScalerMetricsMap instanceof Map) {
- Object pollIdleRatioAvg = ((Map<?, ?>)
autoScalerMetricsMap).get(SeekableStreamIndexTaskRunner.POLL_IDLE_RATIO_KEY);
+ Object pollIdleRatioAvg = ((Map<?, ?>)
autoScalerMetricsMap).get(metricName);
Review Comment:
[P2] Preserve legacy task-stat fallbacks during rolling upgrades
**Finding:** Both extractRunningAvgProcessingRate and extractPollIdleRatio
now read only the new running-average key. During a rolling upgrade, supervisor
stats can mix old tasks exposing movingAverages/pollIdleRatio with new tasks;
old-only reports return -1 and mixed reports are averaged using only new tasks,
causing autoscaling to skip or use incomplete metrics.
**Suggestion:** Fall back to each task's legacy processing-rate or poll-idle
metric when its running-average key is absent, and add mixed-version coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
