Kim,
Any tools to read or dump messages from journal files?
Thanks,
Ram
On Fri, Dec 7, 2018 at 11:32 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
> Kim,
>
> We have only one main queue and one dead letter queue, and we have 10
> produces and 12 consumers and producers pump 1k mess
Kim,
We have only one main queue and one dead letter queue, and we have 10
produces and 12 consumers and producers pump 1k messages/sec. Below is
qpid-stat -q output when it stopped taking any more messages.
bash-4.1# qpid-stat -q
Queues
queue dur autoDel e
This looks like a bug to me, and that is why I am keen to see a
reproducer if you can find one. How many queues are there? How many
producers and consumers are there for each queue? How are the consumers
working? Are they configured as listeners, or do they poll for new
messages? How frequently
Kim,
We have set wcache-page-size=128 in qpidd.conf, restarted broker and let
client recreated the queues fresh, we still getting this error, how do we
find if queues created by client actually have this wcache-page-size=128?
2018-12-05 21:18:16 [Protocol] error Connection
qpid.:5672-:17769 close
Kim,
Thank you, i will play with that setting, please let me know if any other
tunings will help.
Ram
On Wed, Nov 28, 2018 at 8:04 AM Kim van der Riet
wrote:
> The answer to your first question depends on what is more important to
> you - low latency or high throughput. Messages to be persiste
The answer to your first question depends on what is more important to
you - low latency or high throughput. Messages to be persisted will
accumulate in a buffer page until it is full or until a timer is
triggered, then it will be written to disk. It is not until this happens
that the message w
Kim,
1. My message size is around 80kb, so what would be suggested values for
the blow properties?
wcache-page-size
wcache-num-pages
tpl-wcache-num-pages
tpl-wcache-page-size
right now i have all defaults, so i am trying to see if i can tune these
values for my messages size to avoid those AIO
There is little documentation on linearstore. Certainly, the Apache docs
don't contain much. I think this is an oversight, but it won't get fixed
anytime soon.
Kim
On 11/16/18 12:11 PM, rammohan ganapavarapu wrote:
Any one point me to the doc where i can read internals about how
linearstore w
Kim,
Actually we are using qpid as part of our application, and customer is
using my application. They are facing this issue and it is happening to
them intermittently but still don't know at what scenario it is happening.
I was trying the same application with NFS but still couldn't reproduce it.
Any one point me to the doc where i can read internals about how
linearstore works and how qpid uses it?
Thanks,
Ram
On Mon, Nov 12, 2018 at 8:43 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
> Kim,
>
> Thanks for clearing that up for me, does it support SAN storage blocks.
> Where
Did you find a reproducer at all?
Kim
On 11/12/18 11:43 AM, rammohan ganapavarapu wrote:
Kim,
Thanks for clearing that up for me, does it support SAN storage blocks.
Where can i read more about linearstore if i want to know the low level
internals?
Ram
On Mon, Nov 12, 2018 at 8:32 AM Kim van
Kim,
Thanks for clearing that up for me, does it support SAN storage blocks.
Where can i read more about linearstore if i want to know the low level
internals?
Ram
On Mon, Nov 12, 2018 at 8:32 AM Kim van der Riet
wrote:
> The linearstore relies on using libaio for its async disk writes. The
>
The linearstore relies on using libaio for its async disk writes. The
O_DIRECT flag is used, and this requires a block of aligned memory to
serve as a memory buffer for disk write operations. To my knowledge,
this technique only works with local disks and controllers. NFS does not
allow for DMA
But how does NFS will cause this issue, i am interested to see because we
are using NFS (V4 version) in some environments, so wanted to learn tunings
when we use NFS.
Thanks,
Ram
On Fri, Nov 9, 2018 at 6:48 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
> Sorry, i thought it's NFS bu
Sorry, i thought it's NFS but it's actually SAN storage volume.
Thanks,
Ram
On Fri, Nov 9, 2018, 2:10 AM Gordon Sim On 08/11/18 16:56, rammohan ganapavarapu wrote:
> > I was wrong about the NFS for qpid journal files, looks like they are on
> > NFS, so does NFS cause this issue?
>
> Yes, I belie
On 08/11/18 16:56, rammohan ganapavarapu wrote:
I was wrong about the NFS for qpid journal files, looks like they are on
NFS, so does NFS cause this issue?
Yes, I believe it does. What version of NFS are you using?
-
To unsubs
Kim/Gordon,
I was wrong about the NFS for qpid journal files, looks like they are on
NFS, so does NFS cause this issue?
Ram
On Wed, Nov 7, 2018 at 12:18 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
> Kim,
>
> Ok, i am still trying to see what part of my java application is causing
On 11/7/18 3:18 PM, rammohan ganapavarapu wrote:
Kim,
Ok, i am still trying to see what part of my java application is causing
that issue, yes that issue is happening intermittently. Regarding
"JERR_WMGR_ENQDISCONT" error, may be they are chained exceptions from the
previous error JERR_JCNTL_AIO
v 8, 2018 at 11:26 AM Kim van der Riet
wrote:
> Resending, did not show up on the list the first time I sent it...
>
>
>
> Forwarded Message ----
> Subject:Re: qpid-cpp-0.35 errors
> Date: Thu, 8 Nov 2018 09:30:24 -0500
> From: Kim van der Riet
>
Do you have any kernel (net/disk) tuning recommendations for qpid-cpp with
linearstore?
Ram
On Thu, Nov 8, 2018 at 8:56 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
> Kim/Gordon,
>
> I was wrong about the NFS for qpid journal files, looks like they are on
> NFS, so does NFS cause t
Resending, did not show up on the list the first time I sent it...
Forwarded Message
Subject:Re: qpid-cpp-0.35 errors
Date: Thu, 8 Nov 2018 09:30:24 -0500
From: Kim van der Riet
To: users@qpid.apache.org
On 11/7/18 3:18 PM, rammohan ganapavarapu wrote
Kim,
Ok, i am still trying to see what part of my java application is causing
that issue, yes that issue is happening intermittently. Regarding
"JERR_WMGR_ENQDISCONT" error, may be they are chained exceptions from the
previous error JERR_JCNTL_AIOCMPLWAIT?
Does message size contribute to this iss
No, they are not.
These two defines govern the number of sleeps and the sleep time while
waiting for before throwing an exception during recovery only. They do
not play a role during normal operation.
If you are able to compile the broker code, you can try playing with
these values. But I do
Kim,
I see these two settings from code, can these be configurable?
#define MAX_AIO_SLEEPS 10 // tot: ~1 sec
#define AIO_SLEEP_TIME_US 10 // 0.01 ms
Ram
On Wed, Nov 7, 2018 at 7:04 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
> Thank you Kim, i will try your suggestions.
>
Thank you Kim, i will try your suggestions.
On Wed, Nov 7, 2018, 6:58 AM Kim van der Riet This error is a linearstore issue. It looks as though there is a single
> write operation to disk that has become stuck, and is holding up all
> further write operations. This happens because there is a fixe
This error is a linearstore issue. It looks as though there is a single
write operation to disk that has become stuck, and is holding up all
further write operations. This happens because there is a fixed circular
pool of memory pages used for the AIO operations to disk, and when one
of these i
Any help in understand why/when broker throws those errors and stop
receiving message would be appreciated.
Not sure if any kernel tuning or broker tuning needs to be done to
solve this issue.
Thanks in advance,
Ram
On Tue, Nov 6, 2018 at 8:35 AM rammohan ganapavarapu <
rammohanga...@gmail.com>
Also from this log message (store level) it seems like waiting for AIO to
complete.
2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": get_events() returned JERR_JCNTL_AIOCMPLWAIT;
wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF
ps=[-A--]
page_state ps=[
I was check the code and i see these lines for that AIO timeout.
case qpid::linearstore::journal::RHM_IORES_PAGE_AIOWAIT:
if (++aio_sleep_cnt > MAX_AIO_SLEEPS)
THROW_STORE_EXCEPTION("Timeout waiting for AIO in
MessageStoreImpl::recoverMessages()");
Actually we have upgraded from qpid-cpp 0.28 to 1.35 and after that we see
this message
2018-10-27 18:58:25 [Store] warning Linear Store: Journal "":
Bad record alignment found at fid=0x4605b offs=0x107680 (likely journal
overwrite boundary); 19 filler record(s) required.
2018-10-27 18:58:25 [Stor
No, local disk.
On Mon, Nov 5, 2018 at 3:26 PM Gordon Sim wrote:
> On 05/11/18 22:58, rammohan ganapavarapu wrote:
> > Gordon,
> >
> > We are using java client 0.28 version and qpidd-cpp 1.35 version
> > (qpid-cpp-server-1.35.0-1.el7.x86_64), i dont know at what scenario its
> > happening but af
On 05/11/18 22:58, rammohan ganapavarapu wrote:
Gordon,
We are using java client 0.28 version and qpidd-cpp 1.35 version
(qpid-cpp-server-1.35.0-1.el7.x86_64), i dont know at what scenario its
happening but after i restart broker and if we wait for few days its
happening again. From the above lo
Gordon,
We are using java client 0.28 version and qpidd-cpp 1.35 version
(qpid-cpp-server-1.35.0-1.el7.x86_64), i dont know at what scenario its
happening but after i restart broker and if we wait for few days its
happening again. From the above logs do you have any pointers to check?
We are usin
On 05/11/18 20:57, rammohan ganapavarapu wrote:
Actually there are no messages in queue, all they messages got consumed by
consumer.
But it still will not enqueue any further messages? Can you reproduce
this easily?
One other suggestion is to try with the linear store rather than the
legacy
I also see this in qpidd logs for broker,store and protocol. Please help me
to understand what it means? why i am i getting "Timeout waiting for AIOs
to complete" ? does it means some thing wrong with journal files?
2018-02-28 13:19:00 [Store] critical Journal "q-001": get_events() returned
JERR_J
Actually there are no messages in queue, all they messages got consumed by
consumer. I also observe two tcp connections to each client and for this
client only one tcp connection. Why does qpid creates two connections?
Ram
On Mon, Nov 5, 2018, 11:09 AM Gordon Sim Can you drain the queue?
>
> ---
Can you drain the queue?
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
I am using java client version 0.28 and qpid-cpp 1.35 version, and i see
this error on client side(producer).
2018-11-02 00:00:46,376 IoSender - /1.2.3.4:5672 INFO
o.a.q.t.n.io.IoSender - Logger.info() : Exception in thread sending to '/
1.2.3.4:5672': java.net.SocketException: Broken pipe (Write
Hi,
Any one else saw this error before? after this error broker stop taking
any messages, not sure what is causing this error.
Thanks,
Ram
On Fri, Nov 2, 2018 at 4:24 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
> Kim/Gordon,
>
> After this message broker is not accepting any mor
Kim/Gordon,
After this message broker is not accepting any more messages and keep
throwing this message.
Thanks,
Ram
On Fri, Nov 2, 2018 at 8:59 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
> Any help in understating this error message would be appreciated.
>
> Ram
>
> On Wed, Oct
Any help in understating this error message would be appreciated.
Ram
On Wed, Oct 31, 2018 at 5:47 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
> Kim,
>
> Any idea about this error?
>
> Thanks,
> Ram
>
> On Tue, Oct 30, 2018, 2:13 PM Gordon Sim wrote:
>
>> On 30/10/18 18:59, rammo
Kim,
Any idea about this error?
Thanks,
Ram
On Tue, Oct 30, 2018, 2:13 PM Gordon Sim wrote:
> On 30/10/18 18:59, rammohan ganapavarapu wrote:
> > There are two more error from my original post, can some one help me to
> > understand when qpid throws these error?
> >
> >
> > 1. 1. 2018-10-2
Gordon,
Thank you.
On Tue, Oct 30, 2018 at 2:13 PM Gordon Sim wrote:
> On 30/10/18 18:59, rammohan ganapavarapu wrote:
> > There are two more error from my original post, can some one help me to
> > understand when qpid throws these error?
> >
> >
> > 1. 1. 2018-10-22 08:05:30 [Broker] err
On 30/10/18 18:59, rammohan ganapavarapu wrote:
There are two more error from my original post, can some one help me to
understand when qpid throws these error?
1. 1. 2018-10-22 08:05:30 [Broker] error Channel exception:
not-attached: Channel 0 is not attached
(/builddir/build/BUIL
Just to add to this , the store level error corresponding to those above
errors is :
2018-02-28 13:19:00 [Store] critical Journal
"ax-q-axgroup-001-consumer-group-001": get_events() returned
JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=2 pc=45 po=0 aer=1 edac:TFFF
ps=[--A-
There are two more error from my original post, can some one help me to
understand when qpid throws these error?
1. 1. 2018-10-22 08:05:30 [Broker] error Channel exception:
not-attached: Channel 0 is not attached
(/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.c
On 29/10/18 21:45, rammohan ganapavarapu wrote:
I will try to produce, do have any suggestions on what to check in trace
logs?
for qpidd, adding --log-enable trace+:Network to whatever other logging
options you have should do it
---
I will try to produce, do have any suggestions on what to check in trace
logs?
On Fri, Oct 26, 2018 at 2:15 AM Gordon Sim wrote:
> On 26/10/18 00:44, rammohan ganapavarapu wrote:
> > Any idea when do we see this error?
>
> In AMQP 0-10, the connection-close frame allows an informational message
On 26/10/18 00:44, rammohan ganapavarapu wrote:
Any idea when do we see this error?
In AMQP 0-10, the connection-close frame allows an informational message
(replyText) to be included. This must be less than 256 bytes. It sounds
like something is either trying to exceed that, or succeeding an
Any idea when do we see this error?
On Tue, Oct 23, 2018 at 10:42 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
> That error i get it but what is this error i am seeing in client ?
>
>
> 2018-10-23 17:10:57,997 IoReceiver - /10.68.94.134:5672 ERROR
> o.a.q.c.AMQConnectionDelegate_0_
That error i get it but what is this error i am seeing in client ?
2018-10-23 17:10:57,997 IoReceiver - /10.68.94.134:5672 ERROR
o.a.q.c.AMQConnectionDelegate_0_10 - AMQConnectionDelegate_0_10.closed() :
connection exception: conn:5668fdd1
org.apache.qpid.transport.ConnectionException: illegal-a
2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot deliver to
queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on
prod-queue-01: current=[count: 12567, size: 1073741816], max=[count:
100, size: 1073741824]
(/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/brok
Hi,
I am seeing lot of these messages in my qpidd logs and i am not sure why am
i seeing these, can some one explain?
2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot deliver to
queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on
prod-queue-01: current=[count: 1
53 matches
Mail list logo
