I occasionally am getting some batch send errors from the stock async producer.
This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
Is there anyway to check what happens when those batch errors occur?
Or bump up the retry count? (looks like it only did a single retry).
I need the speed of the async producer, but it needs to be reliable (I see a
handful of these a day but in a weather alerting system it only takes missing
one let alone 25 or 100/1000).
Here's a stack trace of one of the errors that I'm seeing.
22:23:39.405 [ProducerSendThread-1824508747] WARN k.p.a.DefaultEventHandler -
Error sending messages, 0 attempts remaining
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
~[na:1.6.0_24]
at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
~[na:1.6.0_24]
at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
~[na:1.6.0_24]
at
kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
~[apns-consumer-1.0.jar:na]
at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
~[apns-consumer-1.0.jar:na]
at
kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
~[apns-consumer-1.0.jar:na]
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
~[apns-consumer-1.0.jar:na]
at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
~[apns-consumer-1.0.jar:na]
at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
~[apns-consumer-1.0.jar:na]
at
kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
[apns-consumer-1.0.jar:na]
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
[apns-consumer-1.0.jar:na]
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
[apns-consumer-1.0.jar:na]
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
[apns-consumer-1.0.jar:na]
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
[apns-consumer-1.0.jar:na]
at scala.collection.immutable.Stream.foreach(Stream.scala:260)
[apns-consumer-1.0.jar:na]
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
[apns-consumer-1.0.jar:na]
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
[apns-consumer-1.0.jar:na]
22:23:39.406 [ProducerSendThread-1824508747] ERROR k.p.a.ProducerSendThread -
Error in handling batch of 27 events
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
~[na:1.6.0_24]
at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
~[na:1.6.0_24]
at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
~[na:1.6.0_24]
at
kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
~[apns-consumer-1.0.jar:na]
at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
~[apns-consumer-1.0.jar:na]
at
kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
~[apns-consumer-1.0.jar:na]
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
~[apns-consumer-1.0.jar:na]
at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
~[apns-consumer-1.0.jar:na]
at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
~[apns-consumer-1.0.jar:na]
at
kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
~[apns-consumer-1.0.jar:na]
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
~[apns-consumer-1.0.jar:na]
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
[apns-consumer-1.0.jar:na]
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
[apns-consumer-1.0.jar:na]
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
[apns-consumer-1.0.jar:na]
at scala.collection.immutable.Stream.foreach(Stream.scala:260)
[apns-consumer-1.0.jar:na]
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
[apns-consumer-1.0.jar:na]
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
[apns-consumer-1.0.jar:na]
