Hello,
We are seeing some strange behaviour with commitOffsets() method of
kafka.javaapi.consumer.ConsumerConnector. We committing the offsets to
zookeeper at the end of the consumer batch. We are running multiple
consumers for the same topic.
Test details:
1. Created a topic with three partitions
2. Started three consumers (cronjob) at the same time. The aim is that each
consumer to process one partition.
3. Each consumer at the end of the batch, it will call the commitOffsets()
method on kafka.javaapi.consumer.ConsumerConnector
4. The offsets are getting properly updated in zookeeper if we run the
consumers for small set (say 1000 messages) of messages.
5. But for larger number of messages, commit offset is not working as
expected…sometimes only two offsets are properly committing and other one
remains as it was.
6. Please see the below example
Partition: 0 Latest Offset: 1057585
Partition: 1 Latest Offset: 1057715
Partition: 2 Latest Offset: 1057590
Earliest Offset after all consumers completed:
{0=1057585, 1=724375, 2=1057590}
Highlighted in red supposed to be committed as 1057715 but it did not.
Please check if it is bug with multiple consumers. When multiple consumers
are trying to update the same path in Zookeper, is there any
synchronization issue?
Kafka Cluster details
1 zookeeper
3 brokers
Thanks
Sreeni
