Alan Conway wrote:
Just got thinking about this - how to ensure work is distributed
efficiently.
Simple case: 1 q many consumers. Automatically balances itself because
consumers request messages at exactly the rate they can consume them.
To scale up we need more qs.
Why do we need extra queues to scale up? Perhaps separate queues could
result in slightly less lock contention but other than that it doesn't
seem to improve scalability unless you moved the separate queues to
different broker processes. Queue length could also be kept down for
each individual queue but the overall resources required wouldn't be any
less. Or am I missing something obvious?
The demo aims for balance by having an
exchange that tries to keep the queues to the same length.
When work is coming in faster than it's being processed the exchange can
balance the queues by adding messages to the short queues.
But what happens during slow periods when work is processed faster than
it comes in? The fast queues are emptying out and there's work trapped
on the slow queues. The exchange can't balance things by adding messages
because there are no new messages to add.
So let the exchange act as a consumer and *remove* messages from
over-full queues to re-queue them on under-full/empty ones. Now we have
a full dynamic balance in both growing and shrinking phases
This complicates the exchange though as it now presumably needs to
periodically monitor the queue lengths (i.e. it becomes an active entity
rather than just reacting to publications routed through it). Maybe an
entirely separate re-balancing component would be cleaner?
This gives us the best balance but it incurs overhead when the broker
has to move message between queues. So perhaps a final optimization is
to have the exchange track not only the fullness of the queues but the
rate at which they empty. Then instead of aiming for every q having the
same length, it can (over time) aim for queues to be filled in
proportion to their throughput. I need to think about that some more
optimization strategies don't always perform the way you expect!
I'm not really sure I understand the root problem here. i.e. why do we
want multiple queues of the same (or similar) length?
