Kanak Biscuitwala created KAFKA-3966:
----------------------------------------
Summary: Consumer briefly ignores partitions on backlogs
Key: KAFKA-3966
URL: https://issues.apache.org/jira/browse/KAFKA-3966
Project: Kafka
Issue Type: Bug
Reporter: Kanak Biscuitwala
Attachments: screenshot-1.png
Setup:
Kafka 0.10.0.0
9 brokers
36 partitions
12 members in the consumer group
5 producers, partitioning data roughly evenly
max.poll.records = 10
~1500 events/sec
~500 bytes/message
KafkaConsumer API
In the following setup, 3 partitions are assigned to each consumer (and 12 are
served by each broker). I've noticed that the third of the three partitions
tends to be ignored in favor of the first two on each consumer, even though the
third partition has data available. Only until the first two partitions are
fully caught up does the consumer return back messages from the third. This
causes a frustrating imbalance in which the same partitions always fall behind.
As a side note, this is true for all of our topics, regardless of partitioning
strategy. The problem goes away if there are exactly as many consumers as
partitions.
I can attach a screenshot showing the same partitions falling behind (verified
that they're each assigned to different nodes), if that is helpful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
