On 25 May 2017 at 00:02, Ramayan Tiwari wrote:
> Hi Keith,
>
> Thanks so much for the update and putting together the new version with
> enhancements to flowToDisk.
No problem at all.
Just to confirm, in order for me to get the
> broker with these fixes and also use MultiQueueConsumer, I will ge
Hi Keith,
Thanks so much for the update and putting together the new version with
enhancements to flowToDisk. Just to confirm, in order for me to get the
broker with these fixes and also use MultiQueueConsumer, I will get 6.0.7
version and apply QPID-7462 on it?
For the longer terms fix (periodic
Hi Ramayan
Further changes have been made on the 6.0.x branch that prevent the
noisy flow to disk messages in the log. The flow to disk on/off
messages we had before didn't really fit and this resulted in the logs
being spammed when either a queue was considered full or memory was
under pressure
Thanks Keith for the update.
- Ramayan
On Mon, May 15, 2017 at 2:35 AM, Keith W wrote:
> Hi Ramayan
>
> We are still looking at our approach to the Broker's flow to disk
> feature in light of the defect you highlighted. We have some work in
> flight this week investigating alternative approach
Hi Ramayan
We are still looking at our approach to the Broker's flow to disk
feature in light of the defect you highlighted. We have some work in
flight this week investigating alternative approaches which I am
hoping will conclude by the end of week. I should be able to update
you then.
Thanks
Hi Alex,
Any update on the fix for this?
QPID-7753 is assigned a fix version for 7.0.0, I am hoping that the fix
will also be back ported to 6.0.x.
Thanks
Ramayan
On Mon, May 8, 2017 at 2:14 AM, Oleksandr Rudyy wrote:
> Hi Ramayan,
>
> Thanks for testing the patch and providing a feedback.
>
>
Hi Ramayan,
Thanks for testing the patch and providing a feedback.
Regarding direct memory utilization, the Qpid Broker caches up to 256MB of
direct memory internally in QpidByteBuffers. Thus, when testing the Broker
with only 256MB of direct memory, the entire direct memory could be cached
and i
Hi Alex,
Thanks for providing the patch. I verified the fix with same perf test, and
it does prevent broker from going OOM, however. DM utilization doesn't get
any better after hitting the threshold (where flow to disk is activated
based on total used % across broker - graph in the link below).
A
Hi Ramayan,
We attached to the QPID-7753 a patch with a work around for 6.0.x branch.
It triggers flow to disk based on direct memory consumption rather than
estimation of the space occupied by the message content. The flow to disk
should evacuate message content preventing running out of direct m
Hi Ramayan,
The high-level plan is currently as follows:
1) Periodically try to compact sparse direct memory buffers.
2) Increase accuracy of messages' direct memory usage estimation to more
reliably trigger flow to disk.
3) Add an additional flow to disk trigger based on the amount of allocate
Hi Lorenz,
Thanks so much for the patch. We have a perf test now to reproduce this
issue, so we did test with 256KB, 64KB and 4KB network byte buffer. None of
these configurations help with the issue (or give any more breathing room)
for our use case. We would like to share the perf analysis with
Hello Ramayan,
We are still working on a fix for this issue.
In the mean time we had an idea to potentially workaround the issue until a
proper fix is released.
The idea is to decrease the qpid network buffer size the broker uses.
While this still allows for sparsely populated buffers it would i
Thanks so much Keith and the team for finding the root cause. We are so
relieved that we fix the root cause shortly.
Couple of things that I forgot to mention on the mitigation steps we took
in the last incident:
1) We triggered GC from JMX bean multiple times, it did not help in
reducing DM alloc
Hello Ramayan
I believe I understand the root cause of the problem. We have
identified a flaw in the direct memory buffer management employed by
Qpid Broker J which for some messaging use-cases can lead to the OOM
direct you describe. For the issue to manifest the producing
application needs to
Hi All,
We have been monitoring the brokers everyday and today we found one
instance where broker’s DM was constantly going up and was about to crash,
so we experimented some mitigations, one of which caused the DM to come
down. Following are the details, which might help us understanding the
issu
Hi Keith,
Thanks so much for your response and digging into the issue. Below are the
answer to your questions:
1) Yeah we are using QPID-7462 with 6.0.5. We couldn't use 6.1 where it was
released because we need JMX support. Here is the destination format:
""%s ; {node : { type : queue }, link :
Hi Ramayan
We have been discussing your problem here and have a couple of questions.
I have been experimenting with use-cases based on your descriptions
above, but so far, have been unsuccessful in reproducing a
"java.lang.OutOfMemoryError: Direct buffer memory" condition. The
direct memory usag
After a lot of log mining, we might have a way to explain the sustained
increased in DirectMemory allocation, the correlation seems to be with the
growth in the size of a Queue that is getting consumed but at a much slower
rate than producers putting messages on this queue.
The pattern we see is t
Another issue that we noticed is when broker goes OOM due to direct memory,
it doesn't create heap dump (specified by "-XX:+HeapDumpOnOutOfMemoryError"),
even when the OOM error is same as what is mentioned in the oracle JVM docs
("java.lang.OutOfMemoryError").
Has anyone been able to find a way t
Alex,
Below are the flow to disk logs from broker having 3million+ messages at
this time. We only have one virtual host. Time is in GMT. Looks like flow
to disk is active on the whole virtual host and not a queue level.
When the same broker went OOM yesterday, I did not see any flow to disk
logs
Ramayan,
Thanks for the details. I would like to clarify whether flow to disk was
triggered today for 3 million messages?
The following logs are issued for flow to disk:
BRK-1014 : Message flow to disk active : Message memory use {0,number,#}KB
exceeds threshold {1,number,#.##}KB
BRK-1015 : Messa
Hi Alex,
Thanks for your response, here are the details:
We use "direct" exchange, without persistence (we specify NON_PERSISTENT
that while sending from client) and use BDB store. We use JSON virtual host
type. We are not using SSL.
When the broker went OOM, we had around 1.3 million messages w
Hi Ramayan,
Could please share with us the details of messaging use case(s) which ended
up in OOM on broker side?
I would like to reproduce the issue on my local broker in order to fix it.
I would appreciate if you could provide as much details as possible,
including, messaging topology, message p
Hi All,
We are using Java broker 6.0.5, with patch to use MultiQueueConsumer
feature. We just finished deploying to production and saw couple of
instances of broker OOM due to running out of DirectMemory buffer
(exceptions at the end of this email).
Here is our setup:
1. Max heap 12g, max direct
24 matches
Mail list logo
