Hello Keith, As stated in this thread[1], we are actually doing some benchmarks to test the performance of the broker and the dispatch router. In the failing test case, I try to send the messages using 4 JMS producers connected directly to the broker each having its own connection (1 JVM per producer). The messages are sent in synchronous mode with AUTO_ACKNOWLEDGE. We try to send them as fast as possible but the test actually takes around 2 hours.
There are no transactions and no batching. Each producer sends a message continuously in the main thread. In order for the test to pass, I had to set the maximum heap to 32 GB which resulted in augmenting the direct memory to 32 GB also. Because the test takes a long time, I have to run it at night. So I cannot experiment different heap sizes now. [1]: http://qpid.2158936.n2.nabble.com/Performance-Benchmarking-Qpid-dispatch-router-0-6-0-with-Qpid-Java-Broker-6-0-0-td7648076.html Regards, Adel > From: keith.w...@gmail.com > Date: Fri, 29 Jul 2016 11:44:57 +0100 > Subject: Re: [Java Broker - 6.0.1] OutOfMemory in the directy memory buffer > To: users@qpid.apache.org > > Hello Adel > > Since v6, the Java Broker keeps messages in direct memory rather than > heap. There is some explanation here: > > https://qpid.apache.org/releases/qpid-java-6.0.4/java-broker/book/Java-Broker-Runtime-Memory.html > > Internally the Broker for Java uses a direct-memory caching strategy > that aims to avoid the underlying JVM having to release/free chunks of > direct memory frequently. In the AMQP 1.0 implementation within the > Broker there are currently some compromises around this caching > strategy that mean it sometimes relies on a JVM GC to free direct > memory. Now, the JVM normally does not trigger a GC until there is > pressure on the _heap_. I speculate that as your heap is so large, > that the GC events are not occurring sufficiently frequently and > meaning direct memory is not released and allowing direct memory to > became exhausted. Try experimenting by reducing the heap. > > If you can tell us a little more about your test case, we may be able > to make other suggestions. How exactly are the 5,000,000 messages > arriving at the Broker? Are they arriving on a short period of time? > How connections are used to publish the messages? Are transactions in > use? Batching? > > > Kind regards, Keith. > > On 28 July 2016 at 10:58, Adel Boutros <adelbout...@live.com> wrote: > > Hello, > > > > While sending messages to the Java broker which is backed by a Berkley DB > > message store, I am getting the below exception. However, the heap memory > > is stable so I assume the problem is coming from elsewhere. > > I am sending 5 000 000 messages which contain a byte array of 100KB of > > data. I have the error at around 4 000 000 messages. Please note that the > > messages are all being consumed by consumers which are not congested. > > > > I have also monitored the heap and it is never above 1 GB. > > > > So I have the following questions: > > What parts of the broker uses direct memory?Does the fact using BDB means > > that each message which is referenced by BDB has memory allocated on the > > direct memory even if the message is consumed? > > How can I analyse this OutOfMemory exception? (There is no heap dump So I > > assume the error is not related to GC and I personally suspect the BDB to > > be the culprit) > > > > > > Broker Console output > > Info: QPID_JAVA_GC not set. Defaulting to JAVA_GC -XX:+UseConcMarkSweepGC > > -XX:+HeapDumpOnOutOfMemoryError > > Using QPID_JAVA_MEM setting -Xmx16g -Xms2g > > [Broker] BRK-1006 : Using configuration : > > /dell445srv/apps/messaging_server/brokers/work1/config.json > > [Broker] BRK-1001 : Startup : Version: 6.0.1 Build: 1731621 > > [Broker] BRK-1010 : Platform : JVM : Oracle Corporation version: > > 1.7.0_80-b15 OS : Linux version: 2.6.32-358.el6.x86_64 arch: amd64 cores: 20 > > [Broker] BRK-1011 : Maximum Memory : Heap : 17,049,059,328 bytes Direct : > > 17,049,059,328 bytes > > [Broker] BRK-1017 : Process : PID : 26179 > > [Broker] BRK-1002 : Starting : Listening on TCP port 10255 > > [Broker] MNG-1001 : Web Management Startup > > [Broker] MNG-1002 : Starting : HTTP : Listening on TCP port 10256 > > [Broker] MNG-1004 : Web Management Ready > > [Broker] BRK-1004 : Qpid Broker Ready > > ######################################################################## > > # > > # Unhandled Exception java.lang.OutOfMemoryError: Direct buffer memory in > > Thread IO-/10.25.6.48:54546 > > # > > # Exiting > > # > > ######################################################################## > > java.lang.OutOfMemoryError: Direct buffer memory > > ... > > > > > > > > Qpid Log > > 2016-07-28 11:41:50,143 ERROR [IO-/10.25.6.48:54546] (o.a.q.s.Main) - > > Uncaught exception, shutting down. > > java.lang.OutOfMemoryError: Direct buffer memory > > at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_80] > > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > > ~[na:1.7.0_80] > > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > > ~[na:1.7.0_80] > > at > > org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:469) > > ~[qpid-common-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:482) > > ~[qpid-common-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.amqp_1_0.framing.FrameHandler.parse(FrameHandler.java:149) > > ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:440) > > ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:436) > > ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1] > > at java.security.AccessController.doPrivileged(Native Method) > > ~[na:1.7.0_80] > > at > > org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0.received(AMQPConnection_1_0.java:435) > > ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:142) > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:564) > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:58) > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:463) > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:259) > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:108) > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:505) > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:338) > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:87) > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > at > > org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:463) > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > ~[na:1.7.0_80] > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > ~[na:1.7.0_80] > > at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80] > > > > > > > > > > > > > > Regards, > > Adel > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org >
