Chris Egerton created KAFKA-13764:
-------------------------------------
Summary: Potential improvements for Connect incremental
rebalancing logic
Key: KAFKA-13764
URL: https://issues.apache.org/jira/browse/KAFKA-13764
Project: Kafka
Issue Type: Improvement
Components: KafkaConnect
Reporter: Chris Egerton
Assignee: Chris Egerton
There are a few small changes that we might make to the incremental rebalancing
logic for Kafka Connect to improve distribution of connectors and tasks across
a cluster and address potential bugs:
# During assignment, assign new connectors and tasks across the cluster before
calculating revocations that may be necessary in order to balance the cluster.
This way, we can potentially skip a round of revocation by using newly-created
connectors and tasks to balance out the cluster.
# Perform connector and task revocation in more cases, such as when one or
more connectors are reconfigured to use fewer tasks, which can possibly lead to
an imbalanced cluster.
# Fix [this
line|https://github.com/apache/kafka/blob/06ca4850c5b2b12e972f48e03fe4f9c1032f9a3e/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/IncrementalCooperativeAssignor.java#L248]
to use the same aggregation logic that's used
[here|https://github.com/apache/kafka/blob/06ca4850c5b2b12e972f48e03fe4f9c1032f9a3e/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/IncrementalCooperativeAssignor.java#L273-L281]
in order to avoid overwriting map values when they should be combined instead.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
