Hi Ramayan, Could please share with us the details of messaging use case(s) which ended up in OOM on broker side? I would like to reproduce the issue on my local broker in order to fix it. I would appreciate if you could provide as much details as possible, including, messaging topology, message persistence type, message sizes,volumes, etc.
Qpid Broker 6.0.x uses direct memory for keeping message content and receiving/sending data. Each plain connection utilizes 512K of direct memory. Each SSL connection uses 1M of direct memory. Your memory settings look Ok to me. Kind Regards, Alex On 18 April 2017 at 23:39, Ramayan Tiwari <ramayan.tiw...@gmail.com> wrote: > Hi All, > > We are using Java broker 6.0.5, with patch to use MultiQueueConsumer > feature. We just finished deploying to production and saw couple of > instances of broker OOM due to running out of DirectMemory buffer > (exceptions at the end of this email). > > Here is our setup: > 1. Max heap 12g, max direct memory 4g (this is opposite of what the > recommendation is, however, for our use cause message payload is really > small ~400bytes and is way less than the per message overhead of 1KB). In > perf testing, we were able to put 2 million messages without any issues. > 2. ~400 connections to broker. > 3. Each connection has 20 sessions and there is one multi queue consumer > attached to each session, listening to around 1000 queues. > 4. We are still using 0.16 client (I know). > > With the above setup, the baseline utilization (without any messages) for > direct memory was around 230mb (with 410 connection each taking 500KB). > > Based on our understanding of broker memory allocation, message payload > should be the only thing adding to direct memory utilization (on top of > baseline), however, we are experiencing something completely different. In > our last broker crash, we see that broker is constantly running with 90%+ > direct memory allocated, even when message payload sum from all the queues > is only 6-8% (these % are against available DM of 4gb). During these high > DM usage period, heap usage was around 60% (of 12gb). > > We would like some help in understanding what could be the reason of these > high DM allocations. Are there things other than message payload and AMQP > connection, which use DM and could be contributing to these high usage? > > Another thing where we are puzzled is the de-allocation of DM byte buffers. > From log mining of heap and DM utilization, de-allocation of DM doesn't > correlate with heap GC. If anyone has seen any documentation related to > this, it would be very helpful if you could share that. > > Thanks > Ramayan > > > *Exceptions* > > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.8.0_40] > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > ~[na:1.8.0_40] > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_40] > at > org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect( > QpidByteBuffer.java:474) > ~[qpid-common-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate. > restoreApplicationBufferForWrite(NonBlockingConnectionPlainDele > gate.java:93) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.NonBlockingConnectionPlainDele > gate.processData(NonBlockingConnectionPlainDelegate.java:60) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.NonBlockingConnection.doRead( > NonBlockingConnection.java:506) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.NonBlockingConnection.doWork( > NonBlockingConnection.java:285) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.NetworkConnectionScheduler. > processConnection(NetworkConnectionScheduler.java:124) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor. > processConnection(SelectorThread.java:504) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.SelectorThread$ > SelectionTask.performSelect(SelectorThread.java:337) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.SelectorThread$SelectionTask.run( > SelectorThread.java:87) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.SelectorThread.run( > SelectorThread.java:462) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > ~[na:1.8.0_40] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > ~[na:1.8.0_40] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_40] > > > > *Second exception* > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.8.0_40] > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > ~[na:1.8.0_40] > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_40] > at > org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect( > QpidByteBuffer.java:474) > ~[qpid-common-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.NonBlockingConnectionPlainDele > gate.<init>(NonBlockingConnectionPlainDelegate.java:45) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.NonBlockingConnection. > setTransportEncryption(NonBlockingConnection.java:625) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.NonBlockingConnection.<init>( > NonBlockingConnection.java:117) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.NonBlockingNetworkTransport. > acceptSocketChannel(NonBlockingNetworkTransport.java:158) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.SelectorThread$SelectionTask$1.run( > SelectorThread.java:191) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > org.apache.qpid.server.transport.SelectorThread.run( > SelectorThread.java:462) > ~[qpid-broker-core-6.0.5.jar:6.0.5] > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > ~[na:1.8.0_40] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > ~[na:1.8.0_40] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_40] >
