[ https://issues.apache.org/jira/browse/KAFKA-13816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520572#comment-17520572 ]
Chris Egerton commented on KAFKA-13816: --------------------------------------- I won't have time to work on this but I have put together a preliminary integration test that reproduces the issue: [https://github.com/C0urante/kafka/commit/a9f119dbf211d33193d6597c0c806b909ba219d2] > Downgrading Connect rebalancing protocol from incremental to eager causes > duplicate task instances > -------------------------------------------------------------------------------------------------- > > Key: KAFKA-13816 > URL: https://issues.apache.org/jira/browse/KAFKA-13816 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect > Reporter: Chris Egerton > Priority: Major > > The rebalancing protocol for a Kafka Connect cluster can be downgraded from > incremental to eager by adding a worker to the cluster with > {{connect.protocol}} set to {{{}eager{}}}, or by stopping an existing worker > in that cluster, reconfiguring it with the new protocol, and restarting it. > When the worker (re)joins the cluster, a rebalance takes place using the > eager protocol, and duplicate task instances are created on the cluster. > This occurs because: > * The leader does not send out an assignment that revokes all connectors and > tasks for the cluster during that round > * Workers do not respond to the downgrade in protocol by revoking all > connectors and tasks that they were running before the rebalance that are not > included in the new assignment they received during the rebalance > It's likely that this bug hasn't surfaced sooner because any subsequent > rebalance should cause all connectors and tasks on all each in the cluster to > be proactively revoked before the worker rejoins the group. > [KIP-415|https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect#KIP415:IncrementalCooperativeRebalancinginKafkaConnect-Compatibility,Deprecation,andMigrationPlan] > provides one way to address this: > {quote}To downgrade your cluster to use protocol version 0 from version 1 or > higher with {{eager}} rebalancing policy what is required is to switch one of > the workers back to {{eager}} mode. > {panel} > {panel} > |{{connect.protocol = eager}}| > Once this worker joins, the group will downgrade to protocol version 0 and > {{eager}} rebalancing policy, with immediately release of resources upon > joining the group. This process will require a one-time double rebalancing, > with the leader detecting the downgrade and first sending a downgraded > assignment with empty assigned connectors and tasks and from then on just > regular downgraded assignments. > {quote} -- This message was sent by Atlassian Jira (v8.20.1#820001)