[
https://issues.apache.org/jira/browse/KAFKA-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418435#comment-16418435
]
Ewen Cheslack-Postava commented on KAFKA-6717:
----------------------------------------------
Are both assigned the partition in the same generation? As consumers join the
group, it will rebalance and change the assignment. If you are just looking at
which consumers are assigned which partitions, it could appear that two of them
are assigned the same partitions at the same time.
I see you are using the NoOpConsumerRebalanceListener. With this, you wouldn't
see when partitions were assigned or revoked. What are you doing to verify that
both consumer instances are assigned the same partitions at the same time?
Without correct handling of partition assignments and revocation, you
definitely could see data processed twice. In fact, without additional steps
taken to ensure no duplicates, *at least once* handling is what Kafka consumers
would normally provide as long as they handle offset commits properly.
> TopicPartition Assined twice to a consumer group for 2 consumer instances
> --------------------------------------------------------------------------
>
> Key: KAFKA-6717
> URL: https://issues.apache.org/jira/browse/KAFKA-6717
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.11.0.1
> Reporter: Yuancheng PENG
> Priority: Major
>
> I'm using \{{StickyAssignor}} for consuming more than 100 topics with certain
> pattern.
> There are 10 consumers with the same group id.
> I expected that topic-partition to be assigned to only one consumer instance.
> However some topic partitions are assigned twice in 2 different difference
> instance, hence the consumer group process duplicate messages.
> {code:java}
> props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
> Collections.singletonList(StickyAssignor.class));
> KafkaConsumer<Record, Record> c = new KafkaConsumer<>(props);
> c.subscribe(Pattern.compile(TOPIC_PATTERN), new
> NoOpConsumerRebalanceListener());
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
