I just fired up a sample app based on the file-size CBR I mentioned, hope I didn't misunderstand or oversimplify so take a look if you haven't solved it yet. https://github.com/levackt/samples/tree/master/file-size-cbr
Also take a look at the route throttling example which shows off the policies nicely. http://camel.apache.org/route-throttling-example.html Taariq On Tue, Dec 30, 2014 at 8:32 PM, Taariq Levack <taar...@gmail.com> wrote: > How about a content based router that checks the file size and forwards > large messages to one endpoint and smaller ones to another? > Both are throttled or limited appropriately, and forward to your existing > endpoint where the work is done. > > That last endpoint should have more consumers than either of the others. > You can get more creative with the general idea, like dynamically setting > the maximumRequestsPerPeriod[1] higher for the small file endpoint when > there are no large files to process. > [1] http://camel.apache.org/throttler.html > > Taariq > > > On 30 Dec 2014, at 13:41, Nathan Jones <nat...@ncjones.com> wrote: > > > > We are trying to improve the responsiveness of some bulk message > processes such that a large batch does not flood a queue and prevent > subsequent smaller batches getting through in a timely fashion. For > example, a job to import millions of records from CSV may take an hour but > a smaller job to import just one thousand records should be able to begin > processing immediately in parallel instead of being sent to the back of a > very long queue. > > > > This seems to me like the sort of thing Camel should be good at but I > have so far not been about to see how this could be achieved. The idea we > have in mind is to have a queue with limited size that will block when it > is full so that the rate of queuing from a large batch would be limited to > consumer capacity. Subsequent batches would have equal opportunity to get > the next message on to the queue. > > > > At first I thought the Camel maxInflightExchanges property could be used > for this but I don't think it has this affect. Is there a way a Camel route > can inspect the size of the target queue to decide whether to suspend or > resume? > > > > Perhaps a message broker can help solve this with either blocking queues > or virtual aggregate queues but I haven't found these in RabbitMQ or > ActiveMQ. > > > > Does anyone have any advice on a way solve this problem with Camel or > otherwise? > > > > - Nathan >
