Gnanasoundari created KAFKA-7114:
------------------------------------
Summary: Kafka Not sending few partitions messages to consumers
Key: KAFKA-7114
URL: https://issues.apache.org/jira/browse/KAFKA-7114
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 1.1.0
Reporter: Gnanasoundari
*Infrastructure:*
Kafka is running in cluster mode with 3 brokers and 3 zookeeper instances.
Kafka broker is running in t2.xlarge AWS instance.
3 Zookeeper instances is running in single t2.xlarge AWS instance.
Kafka Version: 1.1.0
*Issue:*
Kafka is not sending messages to consumer though the consumer is still active
and subscribed to the partition.
*How to reproduce:*
* Continuously send 50,000 records/sec where each record size is about 200
bytes approx. for 24hour duration.
* We have 25 topics with 4 partitions each. All 25 topics are consumed by 4
consumers with auto assignment of partition by kafka.
* Kafka clears all records in partitions except 2 partitions. For two
partitions , only 27 messages are delivered. Remaining messages has not
delivered to consumer.
* During the run kafka brokers cpu utilization is 5%
Problematic partitions details:
|TOPIC|PARTITION|CURRENT-OFFSET|LOG-END-OFFSET|LAG|CONSUMER-ID|HOST|CLIENT-ID|
|topic-di14|3|27|3919844|3919817|consumer-7-dfaf4c6f-5b71-455e-ab12-9984446a9bf4|/10.10.9.22|consumer-7|
|topic-di9|0|27|12231117|12231090|consumer-22-7922babd-8f8b-49db-a111-df523962cca8|/10.10.9.22|consumer-22|
I am not able to see any error in kafka as well as my consumer application.
Consumer is active when i ran ./bin/kafka-consumer-groups.sh command in kafka.
*Kafka Consumer settings:*
enable.auto.commit = true
session.timeout.ms = 90000
heartbeat.interval.ms = 30000
max.poll.records = 1000
poll-interval = 60ms
poll-timeout = 60ms
Please help us to know why this behavior is happening in kafka.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
