Re: Java broker OOM due to DirectMemory

2017-05-25 Thread Keith W
On 25 May 2017 at 00:02, Ramayan Tiwari wrote: > Hi Keith, > > Thanks so much for the update and putting together the new version with > enhancements to flowToDisk. No problem at all. Just to confirm, in order for me to get the > broker with these fixes and also use MultiQueueConsumer, I will ge

Re: Java broker OOM due to DirectMemory

2017-05-24 Thread Ramayan Tiwari
Hi Keith, Thanks so much for the update and putting together the new version with enhancements to flowToDisk. Just to confirm, in order for me to get the broker with these fixes and also use MultiQueueConsumer, I will get 6.0.7 version and apply QPID-7462 on it? For the longer terms fix (periodic

Re: Java broker OOM due to DirectMemory

2017-05-24 Thread Keith W
Hi Ramayan Further changes have been made on the 6.0.x branch that prevent the noisy flow to disk messages in the log. The flow to disk on/off messages we had before didn't really fit and this resulted in the logs being spammed when either a queue was considered full or memory was under pressure

Re: Java broker OOM due to DirectMemory

2017-05-16 Thread Ramayan Tiwari
Thanks Keith for the update. - Ramayan On Mon, May 15, 2017 at 2:35 AM, Keith W wrote: > Hi Ramayan > > We are still looking at our approach to the Broker's flow to disk > feature in light of the defect you highlighted. We have some work in > flight this week investigating alternative approach

Re: Java broker OOM due to DirectMemory

2017-05-15 Thread Keith W
Hi Ramayan We are still looking at our approach to the Broker's flow to disk feature in light of the defect you highlighted. We have some work in flight this week investigating alternative approaches which I am hoping will conclude by the end of week. I should be able to update you then. Thanks

Re: Java broker OOM due to DirectMemory

2017-05-12 Thread Ramayan Tiwari
Hi Alex, Any update on the fix for this? QPID-7753 is assigned a fix version for 7.0.0, I am hoping that the fix will also be back ported to 6.0.x. Thanks Ramayan On Mon, May 8, 2017 at 2:14 AM, Oleksandr Rudyy wrote: > Hi Ramayan, > > Thanks for testing the patch and providing a feedback. > >

Re: Java broker OOM due to DirectMemory

2017-05-08 Thread Oleksandr Rudyy
Hi Ramayan, Thanks for testing the patch and providing a feedback. Regarding direct memory utilization, the Qpid Broker caches up to 256MB of direct memory internally in QpidByteBuffers. Thus, when testing the Broker with only 256MB of direct memory, the entire direct memory could be cached and i

Re: Java broker OOM due to DirectMemory

2017-05-05 Thread Ramayan Tiwari
Hi Alex, Thanks for providing the patch. I verified the fix with same perf test, and it does prevent broker from going OOM, however. DM utilization doesn't get any better after hitting the threshold (where flow to disk is activated based on total used % across broker - graph in the link below). A

Re: Java broker OOM due to DirectMemory

2017-05-04 Thread Oleksandr Rudyy
Hi Ramayan, We attached to the QPID-7753 a patch with a work around for 6.0.x branch. It triggers flow to disk based on direct memory consumption rather than estimation of the space occupied by the message content. The flow to disk should evacuate message content preventing running out of direct m

Re: Java broker OOM due to DirectMemory

2017-04-30 Thread Lorenz Quack
Hi Ramayan, The high-level plan is currently as follows: 1) Periodically try to compact sparse direct memory buffers. 2) Increase accuracy of messages' direct memory usage estimation to more reliably trigger flow to disk. 3) Add an additional flow to disk trigger based on the amount of allocate

Re: Java broker OOM due to DirectMemory

2017-04-28 Thread Ramayan Tiwari
Hi Lorenz, Thanks so much for the patch. We have a perf test now to reproduce this issue, so we did test with 256KB, 64KB and 4KB network byte buffer. None of these configurations help with the issue (or give any more breathing room) for our use case. We would like to share the perf analysis with

Re: Java broker OOM due to DirectMemory

2017-04-28 Thread Lorenz Quack
Hello Ramayan, We are still working on a fix for this issue. In the mean time we had an idea to potentially workaround the issue until a proper fix is released. The idea is to decrease the qpid network buffer size the broker uses. While this still allows for sparsely populated buffers it would i

Re: Java broker OOM due to DirectMemory

2017-04-21 Thread Ramayan Tiwari
Thanks so much Keith and the team for finding the root cause. We are so relieved that we fix the root cause shortly. Couple of things that I forgot to mention on the mitigation steps we took in the last incident: 1) We triggered GC from JMX bean multiple times, it did not help in reducing DM alloc

Re: Java broker OOM due to DirectMemory

2017-04-21 Thread Keith W
Hello Ramayan I believe I understand the root cause of the problem. We have identified a flaw in the direct memory buffer management employed by Qpid Broker J which for some messaging use-cases can lead to the OOM direct you describe. For the issue to manifest the producing application needs to

Re: Java broker OOM due to DirectMemory

2017-04-21 Thread Ramayan Tiwari
Hi All, We have been monitoring the brokers everyday and today we found one instance where broker’s DM was constantly going up and was about to crash, so we experimented some mitigations, one of which caused the DM to come down. Following are the details, which might help us understanding the issu

Re: Java broker OOM due to DirectMemory

2017-04-20 Thread Ramayan Tiwari
Hi Keith, Thanks so much for your response and digging into the issue. Below are the answer to your questions: 1) Yeah we are using QPID-7462 with 6.0.5. We couldn't use 6.1 where it was released because we need JMX support. Here is the destination format: ""%s ; {node : { type : queue }, link :

Re: Java broker OOM due to DirectMemory

2017-04-20 Thread Keith W
Hi Ramayan We have been discussing your problem here and have a couple of questions. I have been experimenting with use-cases based on your descriptions above, but so far, have been unsuccessful in reproducing a "java.lang.OutOfMemoryError: Direct buffer memory" condition. The direct memory usag

Re: Java broker OOM due to DirectMemory

2017-04-20 Thread Ramayan Tiwari
After a lot of log mining, we might have a way to explain the sustained increased in DirectMemory allocation, the correlation seems to be with the growth in the size of a Queue that is getting consumed but at a much slower rate than producers putting messages on this queue. The pattern we see is t

Re: Java broker OOM due to DirectMemory

2017-04-19 Thread Ramayan Tiwari
Another issue that we noticed is when broker goes OOM due to direct memory, it doesn't create heap dump (specified by "-XX:+HeapDumpOnOutOfMemoryError"), even when the OOM error is same as what is mentioned in the oracle JVM docs ("java.lang.OutOfMemoryError"). Has anyone been able to find a way t

Re: Java broker OOM due to DirectMemory

2017-04-19 Thread Ramayan Tiwari
Alex, Below are the flow to disk logs from broker having 3million+ messages at this time. We only have one virtual host. Time is in GMT. Looks like flow to disk is active on the whole virtual host and not a queue level. When the same broker went OOM yesterday, I did not see any flow to disk logs

Re: Java broker OOM due to DirectMemory

2017-04-19 Thread Oleksandr Rudyy
Ramayan, Thanks for the details. I would like to clarify whether flow to disk was triggered today for 3 million messages? The following logs are issued for flow to disk: BRK-1014 : Message flow to disk active : Message memory use {0,number,#}KB exceeds threshold {1,number,#.##}KB BRK-1015 : Messa

Re: Java broker OOM due to DirectMemory

2017-04-19 Thread Ramayan Tiwari
Hi Alex, Thanks for your response, here are the details: We use "direct" exchange, without persistence (we specify NON_PERSISTENT that while sending from client) and use BDB store. We use JSON virtual host type. We are not using SSL. When the broker went OOM, we had around 1.3 million messages w

Re: Java broker OOM due to DirectMemory

2017-04-19 Thread Oleksandr Rudyy
Hi Ramayan, Could please share with us the details of messaging use case(s) which ended up in OOM on broker side? I would like to reproduce the issue on my local broker in order to fix it. I would appreciate if you could provide as much details as possible, including, messaging topology, message p

Java broker OOM due to DirectMemory

2017-04-18 Thread Ramayan Tiwari
Hi All, We are using Java broker 6.0.5, with patch to use MultiQueueConsumer feature. We just finished deploying to production and saw couple of instances of broker OOM due to running out of DirectMemory buffer (exceptions at the end of this email). Here is our setup: 1. Max heap 12g, max direct