Tomas Benc created KAFKA-3843:
---------------------------------
Summary: Endless consuming messages in Kafka cluster
Key: KAFKA-3843
URL: https://issues.apache.org/jira/browse/KAFKA-3843
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 0.9.0.1
Reporter: Tomas Benc
Assignee: Neha Narkhede
We are running Kafka in cluster (3 virtual machines). Kafka is configured
min.insync.replicas = 2 and topics are configured replication factor = 3. This
configuration means, there must be at least 2 brokers of 3 in cluster up and
running to receive any messages. This works as expected.
Our consumers are high level consumers and offsets are commited manually
(auto.commit disabled) and stored in Kafka.
Reproducing the issue:
1. Kafka cluster up and running and receives messages
2. Consumers are disabled (messages in Kafka are in lag)
3. Disable 2 Kafka brokers in cluster
4. Enable consumers
Consumers are consuming messages in batch, and commiting offsets after
processing. But commit offsets fails in Kafka, because of
NotEnoughReplicasException. That is correct. What is not correct, high level
consumer has no idea, that offset are not commited and consumes same messages
again and again.
It would be helpful, that method commitOffsets() in interface
kafka.javaapi.consumer.ConsumerConnector should return some information (return
boolean or throw exception) about this operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
