Re: Rate limit producer to match consumer

2014-12-31 Thread Taariq Levack
I just fired up a sample app based on the file-size CBR I mentioned, hope I didn't misunderstand or oversimplify so take a look if you haven't solved it yet. https://github.com/levackt/samples/tree/master/file-size-cbr Also take a look at the route throttling example which shows off the policies

Rate limit producer to match consumer

2014-12-30 Thread Nathan Jones
We are trying to improve the responsiveness of some bulk message processes such that a large batch does not flood a queue and prevent subsequent smaller batches getting through in a timely fashion. For example, a job to import millions of records from CSV may take an hour but a smaller job to

Re: Rate limit producer to match consumer

2014-12-30 Thread Taariq Levack
How about a content based router that checks the file size and forwards large messages to one endpoint and smaller ones to another? Both are throttled or limited appropriately, and forward to your existing endpoint where the work is done. That last endpoint should have more consumers than