Kafka OOME: Direct buffer memory

[Joseph Francis]

We are running kafka 0.9.0.1 in production and saw these exceptions:


[2016-06-23 22:55:10,239] INFO [KafkaApi-3] Closing connection due to error 
during produce request with correlation id 6 from client id kafka-python with 
ack=0


Topic and partition to exceptions: [xyx,8] -> 
kafka.common.MessageSizeTooLargeException (kafka.server.KafkaApis)


[2016-06-23 22:55:41,917] INFO Scheduling log segment 95455988 for log 
abc_json-7 for deletion. (kafka.log.Log)


[2016-06-23 22:55:41,921] INFO Scheduling log segment 2036034857 for log 
xyz_json-3 for deletion. (kafka.log.Log)


[2016-06-23 22:55:51,112] INFO Rolled new log segment for 'abc_json-7' in 1 ms. 
(kafka.log.Log)


[2016-06-23 22:55:59,411] ERROR Processor got uncaught exception. 
(kafka.network.Processor)


java.lang.OutOfMemoryError: Direct buffer memory


        at java.nio.Bits.reserveMemory(Bits.java:658)


        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)


        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)


        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)


        at sun.nio.ch.IOUtil.read(IOUtil.java:195)


        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)


        at 
org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:108)


        at 
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:97)


        at 
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)


        at 
org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153)


        at 
org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134)


        at org.apache.kafka.common.network.Selector.poll(Selector.java:286)


        at kafka.network.Processor.run(SocketServer.scala:413)


        at java.lang.Thread.run(Thread.java:745)


2 of our brokers were affected and caused them to slowdown.

Prior to seeing these exceptions, we did partition reassignments on the cluster 
and observed a steep decrease in cache usage on the nodes.


Not quite sure of the exact cause of the "Direct buffer memory" exceptions, is 
it simply the case that kafka is receiving too large messages that it cant hold 
in memory?


Thanks,

Joseph