Chris Egerton created KAFKA-13816: ------------------------------------- Summary: Downgrading Connect rebalancing protocol from incremental to eager causes duplicate task instances Key: KAFKA-13816 URL: https://issues.apache.org/jira/browse/KAFKA-13816 Project: Kafka Issue Type: Bug Components: KafkaConnect Reporter: Chris Egerton
The rebalancing protocol for a Kafka Connect cluster can be downgraded from incremental to eager by adding a worker to the cluster with {{connect.protocol}} set to {{{}eager{}}}, or by stopping an existing worker in that cluster, reconfiguring it with the new protocol, and restarting it. When the worker (re)joins the cluster, a rebalance takes place using the eager protocol, and duplicate task instances are created on the cluster. This occurs because: * The leader does not send out an assignment that revokes all connectors and tasks for the cluster during that round * Workers do not respond to the downgrade in protocol by revoking all connectors and tasks that they were running before the rebalance that are not included in the new assignment they received during the rebalance It's likely that this bug hasn't surfaced sooner because any subsequent rebalance should cause all connectors and tasks on all each in the cluster to be proactively revoked before the worker rejoins the group. [KIP-415|https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect#KIP415:IncrementalCooperativeRebalancinginKafkaConnect-Compatibility,Deprecation,andMigrationPlan] provides one way to address this: {quote}To downgrade your cluster to use protocol version 0 from version 1 or higher with {{eager}} rebalancing policy what is required is to switch one of the workers back to {{eager}} mode. {panel} {panel} |{{connect.protocol = eager}}| Once this worker joins, the group will downgrade to protocol version 0 and {{eager}} rebalancing policy, with immediately release of resources upon joining the group. This process will require a one-time double rebalancing, with the leader detecting the downgrade and first sending a downgraded assignment with empty assigned connectors and tasks and from then on just regular downgraded assignments. {quote} -- This message was sent by Atlassian Jira (v8.20.1#820001)