Hello,
I have a Kafka Streams application that is consuming from two topics and
internally aggregating, transforming and joining data. Recently I was doing
a rolling restart of the production Kafka brokers and the following errors
appeared in my Kafka Streams application:
2018-04-12 10:39:13,097 [WARN ] (1-producer)
org.apache.kafka.clients.producer.internals.Sender - [Producer
clientId=<application-id>-f063f0c2-e05f-443a-8ac2-e1298b1a274c-StreamThread-1-producer]
Got error produce response with correlation id 2892774 on
topic-partition
<application-id>-KSTREAM-KEY-SELECT-0000000014-repartition-2, retrying
(9 attempts left). Error: NOT_LEADER_FOR_PARTITION
After the 9 attempts the following error was logged:
2018-04-11 10:39:14,125 [ERROR] (1-producer)
org.apache.kafka.streams.processor.internals.RecordCollectorImpl -
task [0_2] Error sending record (key <key> value <value> timestamp
1523522324694) to topic
<application-id>-KSTREAM-KEY-SELECT-0000000014-repartition due to {};
No more records will be sent and no more offsets will be recorded for
this task.org.apache.kafka.common.errors.NotLeaderForPartitionException:
This server is not the leader for that topic-partition.
Followed by:
2018-04-12 10:39:14,140 [ERROR] (amThread-1)
org.apache.kafka.streams.processor.internals.ProcessorStateManager
- task [0_2] Failed to flush state store
KSTREAM-AGGREGATE-STATE-STORE-0000000003:
org.apache.kafka.streams.errors.StreamsException: task [0_2] Abort
sending since an error caught with a previous record (key <key> value
<value> timestamp 1523522324694) to topic
<application-id>-KSTREAM-KEY-SELECT-0000000014-repartition due to
org.apache.kafka.common.errors.NotLeaderForPartitionException: This
server is not the leader for that topic-partition..
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl$1.onCompletion(RecordCollectorImpl.java:118)
~[arb-arpa.jar:1.4.0.0]
at
org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:204)
~[arb-arpa.jar:1.4.0.0]
at
org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:187)
~[arb-arpa.jar:1.4.0.0]
at
org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:627)
~[arb-arpa.jar:1.4.0.0]
at
org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:596)
~[arb-arpa.jar:1.4.0.0]
at
org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:557)
~[arb-arpa.jar:1.4.0.0]
at
org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:481)
~[arb-arpa.jar:1.4.0.0]
at
org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:74)
~[arb-arpa.jar:1.4.0.0]
at
org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:692)
~[arb-arpa.jar:1.4.0.0]
at
org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101)
~[arb-arpa.jar:1.4.0.0]
at
org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:482)
~[arb-arpa.jar:1.4.0.0]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:474)
~[arb-arpa.jar:1.4.0.0]
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
~[arb-arpa.jar:1.4.0.0]
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
~[arb-arpa.jar:1.4.0.0]
at java.lang.Thread.run(Thread.java:748)
~[na:1.8.0_131]org.apache.kafka.common.errors.NotLeaderForPartitionException:
This server is not the leader for that topic-partition.
The application died and stopped processing any new messages and had to be
restarted. In production there are 5 brokers and the replication factor is
3. The version of kafka streams that I use is 1.0.1 and the kafka version
of the broker is 1.0.1. My question is if this is an expected behavior?
Also is there any way to deal with it?
Regards,
Mihaela Stoycheva
