We are trying to improve the responsiveness of some bulk message
processes such that a large batch does not flood a queue and prevent
subsequent smaller batches getting through in a timely fashion. For
example, a job to import millions of records from CSV may take an hour
but a smaller job to import just one thousand records should be able to
begin processing immediately in parallel instead of being sent to the
back of a very long queue.
This seems to me like the sort of thing Camel should be good at but I
have so far not been about to see how this could be achieved. The idea
we have in mind is to have a queue with limited size that will block
when it is full so that the rate of queuing from a large batch would be
limited to consumer capacity. Subsequent batches would have equal
opportunity to get the next message on to the queue.
At first I thought the Camel maxInflightExchanges property could be used
for this but I don't think it has this affect. Is there a way a Camel
route can inspect the size of the target queue to decide whether to
suspend or resume?
Perhaps a message broker can help solve this with either blocking queues
or virtual aggregate queues but I haven't found these in RabbitMQ or
ActiveMQ.
Does anyone have any advice on a way solve this problem with Camel or
otherwise?
- Nathan
- Rate limit producer to match consumer Nathan Jones
-
