kfaraz commented on code in PR #19646:
URL: https://github.com/apache/druid/pull/19646#discussion_r3505304136
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/indexing/autoscaler/CostBasedAutoScalerIntegrationTest.java:
##########
@@ -398,7 +398,9 @@ private int getCurrentTaskCount(String supervisorId)
final String getSupervisorPath =
StringUtils.format("/druid/indexer/v1/supervisor/%s", supervisorId);
final KafkaSupervisorSpec supervisorSpec =
cluster.callApi().serviceClient().onLeaderOverlord(
mapper -> new RequestBuilder(HttpMethod.GET, getSupervisorPath),
- new TypeReference<>() {}
+ new TypeReference<>()
Review Comment:
Nit: Please revert this
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/WeightedCostFunction.java:
##########
@@ -53,27 +53,28 @@ public class WeightedCostFunction
/**
* Minimum rate of processing for any task in records per second. This is
used
* as a placeholder if avg rate is not available to ensure that cost
computations
- * do not return infinitely large lag recovery times.
+ * do not return infinitely large lag recovery times, at the expense of
underestimating the lag cost.
*/
- static final double MIN_PROCESSING_RATE = 1_000;
+ static final double MIN_PROCESSING_RATE = 5_000;
/**
- * Target idle ratio representing the optimal operating point for the
U-shaped idle cost.
+ * Default target idle ratio representing the optimal operating point for
the U-shaped idle cost.
* At this ratio the idle cost is at its minimum; both lower (risk) and
higher (waste) are penalized.
+ * Configurable via {@link CostBasedAutoScalerConfig#getIdealIdleRatio()}.
*/
static final double IDEAL_IDLE_RATIO = 0.25;
/**
* Penalty magnitude applied when idle ratio is 0 (no safety margin).
* Controls the steepness of the U-shape on the under-provisioning side.
*/
- static final double UNDER_PROVISIONING_PENALTY = 2.0;
+ static final double UNDER_PROVISIONING_PENALTY = 1.0;
/**
* Penalty magnitude applied when idle ratio is 1 (fully wasted capacity).
* Controls the steepness of the U-shape on the over-provisioning side.
*/
- static final double OVER_PROVISIONING_PENALTY = 1.0;
+ static final double OVER_PROVISIONING_PENALTY = 2.5;
Review Comment:
Why change this to 2.5 instead of 2.0?
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/WeightedCostFunction.java:
##########
@@ -53,27 +53,28 @@ public class WeightedCostFunction
/**
* Minimum rate of processing for any task in records per second. This is
used
* as a placeholder if avg rate is not available to ensure that cost
computations
- * do not return infinitely large lag recovery times.
+ * do not return infinitely large lag recovery times, at the expense of
underestimating the lag cost.
*/
- static final double MIN_PROCESSING_RATE = 1_000;
+ static final double MIN_PROCESSING_RATE = 5_000;
/**
- * Target idle ratio representing the optimal operating point for the
U-shaped idle cost.
+ * Default target idle ratio representing the optimal operating point for
the U-shaped idle cost.
* At this ratio the idle cost is at its minimum; both lower (risk) and
higher (waste) are penalized.
+ * Configurable via {@link CostBasedAutoScalerConfig#getIdealIdleRatio()}.
*/
static final double IDEAL_IDLE_RATIO = 0.25;
Review Comment:
```suggestion
static final double DEFAULT_TARGET_IDLE_RATIO = 0.25;
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/CostBasedAutoScaler.java:
##########
@@ -250,13 +250,14 @@ int computeOptimalTaskCount(CostMetrics metrics)
log.info(
"Computing optimal taskCount for supervisor[%s] with metrics:"
- + " currentTaskCount[%d], avgPartitionLag[%.1f],
avgProcessingRate[%.1f],"
- + " pollIdleRatio[%.1f], lagWeight[%.1f], idleWeight[%.1f].",
+ + " currentTaskCount[%d], avgPartitionLag[%.1f],
avgProcessingRate[%.1f], maxProcessingRate[%.1f]"
+ + " idleRatio[%.1f], lagWeight[%.1f], idleWeight[%.1f].",
supervisorId,
currentTaskCount,
metrics.getAvgPartitionLag(),
metrics.getAvgProcessingRate(),
- metrics.getPollIdleRatio(),
+ metrics.getMaxObservedRate(),
+ config.isUsePollIdleRatio() ? metrics.getPollIdleRatio() :
metrics.estimateIdleRatioFromProcessingRate(),
Review Comment:
Let's log both the items separately, as that might help an operator choose
one over the other.
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/WeightedCostFunction.java:
##########
@@ -53,27 +53,28 @@ public class WeightedCostFunction
/**
* Minimum rate of processing for any task in records per second. This is
used
* as a placeholder if avg rate is not available to ensure that cost
computations
- * do not return infinitely large lag recovery times.
+ * do not return infinitely large lag recovery times, at the expense of
underestimating the lag cost.
*/
- static final double MIN_PROCESSING_RATE = 1_000;
+ static final double MIN_PROCESSING_RATE = 5_000;
Review Comment:
Why this change?
Technically, 1000 was also an arbitrary number but 5000 definitely seems to
be on the higher side, especially when dealing with bulkier records with large
(say JSON) column values. I would rather the user choose whether they want to
prefer lag recovery or throughput by tweaking the weights.
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/CostBasedAutoScalerConfig.java:
##########
@@ -86,6 +87,7 @@ public CostBasedAutoScalerConfig(
@Nullable @JsonProperty("scaleActionPeriodMillis") Long
scaleActionPeriodMillis,
@Nullable @JsonProperty("lagWeight") Double lagWeight,
@Nullable @JsonProperty("idleWeight") Double idleWeight,
+ @Nullable @JsonProperty("idealIdleRatio") Double idealIdleRatio,
Review Comment:
Maybe rename to `optimalIdleRatio` or `optimalTaskIdleRatio`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
