[
https://issues.apache.org/jira/browse/KAFKA-9468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022513#comment-17022513
]
Ewen Cheslack-Postava commented on KAFKA-9468:
----------------------------------------------
[~rhauch] Allowing things to run in that environment leads to incorrect (and
likely unrecoverable, without a ton of manual effort) state. I'd consider it a
bug, which doesn't need a KIP. The other consideration is potential fallout. It
seems hard to believe someone would be running for very long without running
into problems caused by having more than one partition given at least on
restart/worker addition, you bulk load that data and could easily end up with
misordered data across multiple reconfigurations for the same connector.
> config.storage.topic partition count issue is hard to debug
> -----------------------------------------------------------
>
> Key: KAFKA-9468
> URL: https://issues.apache.org/jira/browse/KAFKA-9468
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 1.0.2, 1.1.1, 2.0.1, 2.1.1, 2.2.2, 2.4.0, 2.3.1
> Reporter: Evelyn Bayes
> Priority: Minor
>
> When you run connect distributed with 2 or more workers and
> config.storage.topic has more then 1 partition, you can end up with one of
> the workers rebalancing endlessly:
> [2020-01-13 12:53:23,535] INFO [Worker clientId=connect-1,
> groupId=connect-cluster] Current config state offset 37 is behind group
> assignment 63, reading to end of config log
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
> [2020-01-13 12:53:23,584] INFO [Worker clientId=connect-1,
> groupId=connect-cluster] Finished reading to end of log and updated config
> snapshot, new config log offset: 37
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
> [2020-01-13 12:53:23,584] INFO [Worker clientId=connect-1,
> groupId=connect-cluster] Current config state offset 37 does not match group
> assignment 63. Forcing rebalance.
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
>
> In case any person viewing this doesn't know you are only ever meant to
> create this topic with one partition.
>
> *Suggested Solution*
> Make the connect worker check the partition count when it starts and if
> partition count is > 1 Kafka Connect stops and logs the reason why.
> I think this is reasonable as it would stop users just starting out from
> building it incorrectly and would be easy to fix early. For those upgrading
> this would easily be caught in a PRE-PROD environment. And even if they
> upgraded directly in PROD you would only be impacted if upgraded all connect
> workers at the same time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
