On 10/12/2020 11:24, Gordon Sim wrote:
On 10/12/2020 09:36, Toralf Lund wrote:
On 09/12/2020 22:55, Gordon Sim wrote:
Does the queue depth on the queue(s) that the slow sends occur on
ever get large? (Is there some limit set on the queue depth?)
The queues are last-value queues sorted on the "subject" values. They
don't grow large, but obviously, some house-keeping is involved when
storing messages.
This leads to another question: When exactly does the broker
acknowledge message reception? Will it actually update queues first?
I mean, it might conceivably read the message, buffer it,
acknowledge, then do the routing.
The broker will only acknowledge the message once it has been
successfully enqueued on all relevant queues. If persistence is in use
then this will be once the journal has been updated on disk.
OK
A last-value queue update involves dequeuing the old message that
matches the key then enqueuing the updated one.
Is the client on which you observe this the only one to publish to a
particular queue? (If not then it seems less likely that the queue is
the issue, if other clients never see such delays).
It should be the only one. I see this for two different queues with a
similar configuration. They are generally written by one application,
and could be read in browse mode by multiple ones.
The only other thing I can think of that would cause the invocation
of send() to take a long time would be some sort of locking issue.
Are the multiple threads possibly sending concurrently?
There is another thread linked to QPid, but it takes care of
"request-response" messaging triggered by user actions. I can't see
how it would be active in all cases where I see delays.
But obviously, it seems quite likely that this is the case of a
thread waiting for *something*. Is there anything else that may
somehow cause a block?
The two things I can think of are waiting for a lock or waiting for
capacity. Waiting for capacity on 0-10 involves a session sync
request. If the session is in use by other threads that could include
those other operations as well.
Well, none of the application threads should use the same session. Like
I said, there is a 2nd messaging thread, but it has a separate
connection and session. On the other hand, I have multiple senders and
receivers on the same session, so perhaps a different thread set up by
the library might be accessing it?
Note that my application has extra threads that do unrelated work.
Also, I expect other threads started by the QPid library will be
active, as there is a receiver with a somewhat large capacity linked
to the same session. And my request-response thread will probably be
listening for incoming messages, too.
I should perhaps have mentioned that there is another problem that I
think may be related: It looks like heartbeat messages are also
sometimes delayed in a similar manner, unless they are lost
completely. Every few hours, I get a reconnect attempt during send or
receive, even though there is probably nothing wrong with the
connection. That leads to a "session busy" exception.
Is there anything different about the case where you see this? (I'm
assuming that not all clients have the same issue and that those that
do are the same ones i.e. on the same machines and with the same
context).
I actually see signs of lost heartbeats in at least one other client. It
accesses different queues and exchanges, but connects to the same
broker, and has as similar layout in some other respects.
Maybe this is where I should start looking. Where and when exactly are
the heartbeats sent? What might cause problems to occur?
If heartbeats are also delayed it seems to me less likely to be
related to the queue or the acknowledgements being delayed.
Again I wonder whether there is anything different about the
installation where this occurs?
That's the million dollar question, I think...
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
