[
https://issues.apache.org/jira/browse/KAFKA-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16421291#comment-16421291
]
Yuancheng PENG commented on KAFKA-6717:
---------------------------------------
Hi, I noticed this by the jmx metrics: the number of messages consumed by this
consumer group is higher than the number of messages sent to those topics. I
also had metrics that report the topic/partition being consumed by each
consumer instance. I don't really have things to do by implementing
RebalanceListener, that's why I left it to NoOpConsumerRebalanceListener. When
a new rebalance occurs, there's a log in each instance saying that the newly
assigned partitions. that's also where I confirmed that some of the topic
partitions are assigned twice.
The issue has occurred quite a few times in our production within few days. I
had to stop and restart all the consumers again to rebalance the topics. (we
have 180+ topics, all topics are single partition, then 10 consumers that share
those topics.).
I change to RoundRobinAssignor recently and this problem does not appear again.
I think this might be an issue of StickyAssignor.
> TopicPartition Assined twice to a consumer group for 2 consumer instances
> --------------------------------------------------------------------------
>
> Key: KAFKA-6717
> URL: https://issues.apache.org/jira/browse/KAFKA-6717
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.11.0.1
> Reporter: Yuancheng PENG
> Priority: Major
>
> I'm using \{{StickyAssignor}} for consuming more than 100 topics with certain
> pattern.
> There are 10 consumers with the same group id.
> I expected that topic-partition to be assigned to only one consumer instance.
> However some topic partitions are assigned twice in 2 different difference
> instance, hence the consumer group process duplicate messages.
> {code:java}
> props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
> Collections.singletonList(StickyAssignor.class));
> KafkaConsumer<Record, Record> c = new KafkaConsumer<>(props);
> c.subscribe(Pattern.compile(TOPIC_PATTERN), new
> NoOpConsumerRebalanceListener());
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
