[
https://issues.apache.org/jira/browse/KAFKA-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16007746#comment-16007746
]
huxi commented on KAFKA-5221:
-----------------------------
Do you mean all these 38,000 partitions reside on a single broker machine? Is
it possible to hit any resource limit?
> Kafka Consumer Attached to partition but not consuming messages
> ---------------------------------------------------------------
>
> Key: KAFKA-5221
> URL: https://issues.apache.org/jira/browse/KAFKA-5221
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.10.2.0
> Environment: Amazon-Linux AMI
> Hardware Specifications - (8 core, 16 GB RAM , 1 TB HardDisk)
> Reporter: Abhimanyu Nagrath
>
> I have a single node Kafka broker and a single zookeeper (3.4.9). I am using
> new Kafka Consumer APIs. One strange thing I observed is when I am starting
> multiple Kafka consumers for multiple topics placed in a single group and on
> hitting ./kafka-consumer-groups.sh this script for the group. Few of the
> consumers are attached to the group but they do not consume any message.
> Below are the stats of group command.
> TOPIC PARTITION CURRENT-OFFSET
> LOG-END-OFFSET LAG CONSUMER-ID HOST
> topic1 0 288 288 0
> consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
> consumer-8
>
> topic1 1 283 283 0
> consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
> consumer-8
>
> topic1 2 279 279 0
> consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
> consumer-8
>
> topic2 0 - 9 -
> consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip>
> consumer-1
>
> topic2 1 - 2 -
> consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip>
> consumer-1
>
> topic3 0 450 450 0
> consumer-3-63c07703-17d0-471b-8c5f-17347699f108 /<serverip>
> consumer-3
>
> topic4 1 - 54
> - consumer-2-94dcc209-8377-45ce-8473-9ab0d85951c4
> /<serverip>
>
> topic2 2 441 441 0
> consumer-5-bcfffc99-5915-41f4-b3e4-970baa204c14 /<serverip>
> So can someone help me that why for topic **topic2** partition 0
> **current-offset** is showing - and **lag** is showing - but messages are
> still there in the server as **LOG-END-OFFSET** is showing 9.
> I am having 600 GB Data distributed among 380 topics and each topic is having
> approx. 100 partitions.
> This is happening very frequently and restarting the consumers solves the
> issue temporarily.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
