Abhimanyu Nagrath created KAFKA-5221:
----------------------------------------
Summary: Kafka Consumer Attached to partition but not consuming
messages
Key: KAFKA-5221
URL: https://issues.apache.org/jira/browse/KAFKA-5221
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 0.10.2.0
Environment: Amazon-Linux AMI
Hardware Specifications - (8 core, 16 GB RAM , 1 TB HardDisk)
Reporter: Abhimanyu Nagrath
I have a single node Kafka broker and a single zookeeper (3.4.9). I am using
new Kafka Consumer APIs. One strange thing I observed is when I am starting
multiple Kafka consumers for multiple topics placed in a single group and on
hitting ./kafka-consumer-groups.sh this script for the group. Few of the
consumers are attached to the group but they do not consume any message.
Below are the stats of group command.
TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID HOST
topic1 0 288 288 0
consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
consumer-8
topic1 1 283 283 0
consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
consumer-8
topic1 2 279 279 0
consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
consumer-8
topic2 0 - 9 -
consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip>
consumer-1
topic2 1 - 2 -
consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip>
consumer-1
topic3 0 450 450 0
consumer-3-63c07703-17d0-471b-8c5f-17347699f108 /<serverip>
consumer-3
topic4 1 - 54
- consumer-2-94dcc209-8377-45ce-8473-9ab0d85951c4 /<serverip>
topic2 2 441 441 0
consumer-5-bcfffc99-5915-41f4-b3e4-970baa204c14 /<serverip>
So can someone help me that why for topic **topic2** partition 0
**current-offset** is showing - and **lag** is showing - but messages are still
there in the server as **LOG-END-OFFSET** is showing 9.
I am having 600 GB Data distributed among 380 topics and each topic is having
approx. 100 partitions.
This is happening very frequently and restarting the consumers solves the issue
temporarily.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
