cryptoe commented on code in PR #16889:
URL: https://github.com/apache/druid/pull/16889#discussion_r1981335039
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/ImmutableWorkerInfo.java:
##########
@@ -225,6 +244,13 @@ private int getWorkerParallelIndexCapacity(double
parallelIndexTaskSlotRatio)
return workerParallelIndexCapacity;
}
+ public Map<String, Integer> incrementTypeSpecificCapacity(String type, int
capacityToAdd)
Review Comment:
Should there be a corresponding decrement ?
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/LimiterUtils.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.setup;
+
+import org.apache.druid.indexing.common.task.Task;
+
+import java.util.Map;
+
+public class LimiterUtils
Review Comment:
```suggestion
public class TaskLimiterUtils
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/LimiterUtils.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.setup;
+
+import org.apache.druid.indexing.common.task.Task;
+
+import java.util.Map;
+
+public class LimiterUtils
+{
+
+ public static boolean canRunTask(Task task, TaskLimits taskLimits, Integer
currentSlotsUsed, Integer totalCapacity)
Review Comment:
This method should have some java docs.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/ImmutableWorkerInfo.java:
##########
@@ -57,6 +61,7 @@ public ImmutableWorkerInfo(
@JsonProperty("worker") Worker worker,
@JsonProperty("currCapacityUsed") int currCapacityUsed,
@JsonProperty("currParallelIndexCapacityUsed") int
currParallelIndexCapacityUsed,
+ @JsonProperty("currCapacityUsedByTaskType") Map<String, Integer>
currCapacityUsedByTaskType,
Review Comment:
or this should be immutable no ?
##########
docs/configuration/index.md:
##########
@@ -1383,6 +1387,16 @@ field. If not provided, the default is to not use it at
all.
|`categoryMap`|A JSON map object mapping a task type String name to a
[CategoryConfig](#categoryconfig) object, by which you can specify category
config for different task type.|`{}`|
|`strong`|With weak workerCategorySpec (the default), tasks for a dataSource
may be assigned to other Middle Managers if the Middle Managers specified in
`categoryMap` are not able to run all pending tasks in the queue for that
dataSource. With strong workerCategorySpec, tasks for a dataSource will only
ever be assigned to their specified Middle Managers, and will wait in the
pending queue if necessary.|false|
+###### taskLimits
+
+The `taskLimits` field can be used with the `equalDistribution`,
`fillCapacity`, `equalDistributionWithCategorySpec` and
`fillCapacityWithCategorySpec` strategies.
+If you don't provide it, it will default to not being used.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`taskLimits`|A map where each key is a task type (`String`), and the
corresponding value represents the absolute limit on the number of task slots
that tasks of this type can occupy. The value is an `Integer` that is greater
than or equal to 0. For example, a value of 5 means that tasks of this type can
occupy up to 5 task slots in total. If both absolute and ratio limits are
specified for the same task type, the effective limit will be the smaller of
the absolute limit and the limit derived from the corresponding ratio.
`taskLimits = {"index_parallel": 3, "query_controller": 5}`. In this example,
parallel indexing tasks can occupy up to 3 task slots, and query controllers
can occupy up to 5 task slots.|`{}`|
+|`taskRatios`|A map where each key is a task type (`String`), and the
corresponding value is a `Double` which should be in the range [0, 1],
representing the ratio of available task slots that tasks of this type can
occupy. This ratio defines the proportion of total task slots a task type can
use, calculated as `ratio * totalSlots`. If both absolute and ratio limits are
specified for the same task type, the effective limit will be the smaller of
the absolute limit and the limit derived from the corresponding ratio.
`taskRatios = {"index_parallel": 0.5, "query_controller": 0.25}`. In this
example, parallel indexing tasks can occupy up to 50% of the total task slots,
and query controllers can occupy up to 25% of the total task slots.|`{}`|
Review Comment:
```suggestion
|`taskRatios`|A map where each key is a task type (`String`), and the
corresponding value is a `Double` which should be in the range [0, 1],
representing the ratio of task slots that tasks of this type can occupy. This
ratio defines the proportion of total task slots a task type can use,
calculated as `ratio * totalSlots`. If both absolute and ratio limits are
specified for the same task type, the effective limit will be the smaller of
the absolute limit and the limit derived from the corresponding ratio.
`taskRatios = {"index_parallel": 0.5, "query_controller": 0.25}`. In this
example, parallel indexing tasks can occupy up to 50% of the total task slots,
and query controllers can occupy up to 25% of the total task slots.|`{}`|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
