We see multiple instance of this error
2017-12-23 05:30:53,722 WARN
org.apache.kafka.clients.producer.internals.Sender: Got error produce
response with correlation id 41987642 on topic-partition events-14,
retrying (2147482899 attempts left). Error: NOT_LEADER_FOR_PARTITION
Followed by this error (repeating multilpe times)
2017-12-23 05:31:09,165 INFO
org.apache.kafka.clients.producer.KafkaProducer: Proceeding to force close
the producer since pending requests could not be completed within timeout 0
ms.
2017-12-23 05:31:09,165 ERROR
org.apache.kafka.clients.producer.internals.ErrorLoggingCallback: Error
when sending message to topic events with key: 36 bytes, value: 8794 bytes
with error: Batch Expired
Once in a while I also notice following error happens only once or two
times between restarts of mm.
2017-09-10 20:02:47,723 FATAL kafka.tools.MirrorMaker$MirrorMakerThread:
[mirrormaker-thread-14] Mirror maker thread failure due to
java.lang.IllegalStateException: Memory records is not writable
at
org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
at
org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:166)
at
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:438)
at
kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:628)
at
kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$6.apply(MirrorMaker.scala:423)
at
kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$6.apply(MirrorMaker.scala:423)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:423)
2017-09-10 20:02:47,728 INFO kafka.tools.MirrorMaker$MirrorMakerThread:
[mirrormaker-thread-14] Flushing producer.
2017-09-10 20:02:47,780 WARN
org.apache.kafka.clients.producer.internals.Sender: Got error produce
response with correlation id 20892 on topic-partition
thirdpartycalls_ingestion_am1-2, retrying (2147483645 attempts left).
Error: NOT_LEADER_FOR_PARTITION
followed by
Clean shutdown of mirror maker
We're using CDH, Kafka 0.9
Can someone explains the reason/meaning of the errors we see ? This will
help us debug this issue further. My understanding is that the producer
threads are failing because for some reason partition leader is changing at
run time. We also noticed that this happens on the perticular day of the
week but I coudln't locate any weekly scheduler in any configuration.
Any pointers will be helpful
Thanks,
Sunil Parmar
