[ https://issues.apache.org/jira/browse/KAFKA-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192827#comment-17192827 ]
BDeus commented on KAFKA-10413: ------------------------------- I have this issue too when EC2 scale in (S3 sink connector). It seems that using the old connect.protocol=eager result in a well balanced tasks across the workers. [https://stackoverflow.com/questions/63348308/how-to-get-kafka-connect-to-balance-tasks-connectors-evenly] Here is a screenshot of tasks monitoring by EC2 !connect_worker_balanced.png! > rebalancing leads to unevenly balanced connectors > ------------------------------------------------- > > Key: KAFKA-10413 > URL: https://issues.apache.org/jira/browse/KAFKA-10413 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect > Affects Versions: 2.5.1 > Reporter: yazgoo > Priority: Major > Attachments: connect_worker_balanced.png > > > Hi, > With CP 5.5, running kafka connect s3 sink on EC2 whith autoscaling enabled, > if a connect instance disappear, or a new one appear, we're seeing unbalanced > consumption, much like mentionned in this post: > [https://stackoverflow.com/questions/58644622/incremental-cooperative-rebalancing-leads-to-unevenly-balanced-connectors] > This usually leads to one kafka connect instance taking most of the load and > consumption not being able to keep on. > Currently, we're "fixing" this by deleting the connector and re-creating it, > but this is far from ideal. > Any suggestion on what we could do to mitigate this ? -- This message was sent by Atlassian Jira (v8.3.4#803005)