Andy Li wrote:
Hi,
What is the best way for the producer to recover (after a pause) and
continue sending messages after "resource-limit-exceeded" exception, when
the queue on the broker fills up? Is it necessary to close/reopen
connection, as it seems?
Creating a new session should be sufficient.
With async sending, how should we pick a point in
the message sequence where we start re-sending messages, to guarantee no
message loss? Can we do better than the last successfull "sync()" point? Is
there a way to restart at the exact interruption spot, to avoid duplicate
message delivery?
Not at present, no. Note that you can also flush instead of syncing;
this requests completion information to be sent but doesn't require
halting the published stream until it arrives (and therefore might allow
at least a smaller in-doubt window). (There is a utility,
qpid::client::MessageReplayTracker used in the failover example that
does this).
There is a message number in the exception sting returned
from the broker. Could it be used to correlate back to the first discarded
message?
There is a command-id, which should correspond back to the message (but
other AMQP 'commands', queueDeclare, messageAccept etc would be in the
same sequence).
This number does not reset after reconnection, and seems to be
per-exchange rather than per-session. And it seems to be vary between X+1
and X+2, where X is 0 -base number of the last message that the subscribers
receive.
In general, would this mechanism be safe/stable to use with some kind of
exponential backoff mechanism?
I'm not clear what you are proposing here.
Or could it cause corruption of the exchange
or broker crash?
Currently, we are using a fanout exchange, with some queue-size-limit and
the "reject" policy. The subscribers have unlimited "prefetch" size for the
LocalQueue (no flow control quota/window) and ACCEPT_MODE_NONE - to free up
broker resources ASAP. However, the queue still overflows after a
sufficiently long publishing burst due to 10X fanout and the resulting
bandwidth disparity per-client between the publisher and each subscriber.
(You can easily see this in the vanilla "perftest" with --mode fanout
--nsubs 10, if you leave "default-queue-limit" setting in the broker, and
run perftest on a separate machine).
Are there any alternative methods of resource-usage-driven publisher flow
control available?
Not yet, but we definitely want to add this.
For example, in ActiveMQ the async publisher is
automatically throttled through the flow control of the underlying TCP
transport.
I've read about the "max-session-rate" option. However, it's static. It
would have to be set up for the worst possible case (all potential
publishers sending in a burst concurrently) if we cannot handle queue
overflow. It would limit the normal-case performance (single publisher
sending a short burst that can fit in the queue until delivered). If we
could throttle the publisher only if the buffer overflows, it seems we would
achieve much better throughput.
Here's our current publisher code:
for(size_t i = 0; i < MAX; i++)
{
//set up msg, stamping with "i"
try
{
async(session).messageTransfer(arg::content=msg,
arg::destination="amq.fanout");
if(i % syncInterval == 0)
session.sync();
}
catch (const qpid::framing::ResourceLimitExceededException& e) {
//wait some time to allow some queue space to free, then restart
connection
//where to roll back "i"?
}
}
Any help you can provide will be greatly appreciated.
Thanks,
Andy
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscr...@qpid.apache.org
