Hi All, We are using Java broker 6.0.5, with patch to use MultiQueueConsumer feature. We just finished deploying to production and saw couple of instances of broker OOM due to running out of DirectMemory buffer (exceptions at the end of this email).
Here is our setup: 1. Max heap 12g, max direct memory 4g (this is opposite of what the recommendation is, however, for our use cause message payload is really small ~400bytes and is way less than the per message overhead of 1KB). In perf testing, we were able to put 2 million messages without any issues. 2. ~400 connections to broker. 3. Each connection has 20 sessions and there is one multi queue consumer attached to each session, listening to around 1000 queues. 4. We are still using 0.16 client (I know). With the above setup, the baseline utilization (without any messages) for direct memory was around 230mb (with 410 connection each taking 500KB). Based on our understanding of broker memory allocation, message payload should be the only thing adding to direct memory utilization (on top of baseline), however, we are experiencing something completely different. In our last broker crash, we see that broker is constantly running with 90%+ direct memory allocated, even when message payload sum from all the queues is only 6-8% (these % are against available DM of 4gb). During these high DM usage period, heap usage was around 60% (of 12gb). We would like some help in understanding what could be the reason of these high DM allocations. Are there things other than message payload and AMQP connection, which use DM and could be contributing to these high usage? Another thing where we are puzzled is the de-allocation of DM byte buffers. >From log mining of heap and DM utilization, de-allocation of DM doesn't correlate with heap GC. If anyone has seen any documentation related to this, it would be very helpful if you could share that. Thanks Ramayan *Exceptions* java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.8.0_40] at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.8.0_40] at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_40] at org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:474) ~[qpid-common-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.restoreApplicationBufferForWrite(NonBlockingConnectionPlainDelegate.java:93) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:60) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:506) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:285) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:124) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:504) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:337) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:87) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:462) ~[qpid-broker-core-6.0.5.jar:6.0.5] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_40] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_40] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_40] *Second exception* java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.8.0_40] at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.8.0_40] at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_40] at org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:474) ~[qpid-common-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.<init>(NonBlockingConnectionPlainDelegate.java:45) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.NonBlockingConnection.setTransportEncryption(NonBlockingConnection.java:625) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.NonBlockingConnection.<init>(NonBlockingConnection.java:117) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.NonBlockingNetworkTransport.acceptSocketChannel(NonBlockingNetworkTransport.java:158) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.SelectorThread$SelectionTask$1.run(SelectorThread.java:191) ~[qpid-broker-core-6.0.5.jar:6.0.5] at org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:462) ~[qpid-broker-core-6.0.5.jar:6.0.5] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_40] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_40] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_40]
