[
https://issues.apache.org/jira/browse/KAFKA-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192827#comment-17192827
]
BDeus edited comment on KAFKA-10413 at 9/9/20, 12:23 PM:
---------------------------------------------------------
I have this issue too when EC2 scale in (S3 sink connector).
It seems that using the old connect.protocol=eager result in a well balanced
tasks across the workers.
[https://stackoverflow.com/questions/63348308/how-to-get-kafka-connect-to-balance-tasks-connectors-evenly]
Here is a screenshot of tasks monitoring by EC2 with scale in at 19h30
!connect_worker_balanced.png!
was (Author: baz33):
I have this issue too when EC2 scale in (S3 sink connector).
It seems that using the old connect.protocol=eager result in a well balanced
tasks across the workers.
[https://stackoverflow.com/questions/63348308/how-to-get-kafka-connect-to-balance-tasks-connectors-evenly]
Here is a screenshot of tasks monitoring by EC2 !connect_worker_balanced.png!
> rebalancing leads to unevenly balanced connectors
> -------------------------------------------------
>
> Key: KAFKA-10413
> URL: https://issues.apache.org/jira/browse/KAFKA-10413
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 2.5.1
> Reporter: yazgoo
> Priority: Major
> Attachments: connect_worker_balanced.png
>
>
> Hi,
> With CP 5.5, running kafka connect s3 sink on EC2 whith autoscaling enabled,
> if a connect instance disappear, or a new one appear, we're seeing unbalanced
> consumption, much like mentionned in this post:
> [https://stackoverflow.com/questions/58644622/incremental-cooperative-rebalancing-leads-to-unevenly-balanced-connectors]
> This usually leads to one kafka connect instance taking most of the load and
> consumption not being able to keep on.
> Currently, we're "fixing" this by deleting the connector and re-creating it,
> but this is far from ideal.
> Any suggestion on what we could do to mitigate this ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
