Re: Curious routing case

Hi Artur, As your group set is unbounded, you may need to look at creating (in-memory) queues and routes at runtime and attaching them to a camel context (or spawning their own context). You can use some camel trickery with DefaultCamelContext + RouteDefinition, it's ugly but it should work. Rega

Re: Curious routing case

If you use ActiveMQ you can use its message grouping support http://activemq.apache.org/message-groups.html And Apache Kafka has topics where you can use partition ids to group by A, B, C and then have parallel processing of each group. Not sure SQS has such advanced features. Its a bit brittle

Re: Curious routing case

I guess one thing that come to my mind is to hide all this parallel stuff inside a processor, that would just spit out on the other end the result of processing all those messages. It would handle grouping and serializing and stuff I guess that would reduce the complexity of the route with a cost o

Re: Curious routing case

Hello, I don't think this route definition is fitting my use case, though I learnt a thing or two about the interesting patterns linked. Thanks! Ok, so let me try to clarify the use case. 1. The stream is infinite, it's not a batch job. The messages keep on coming from SQS 'all the time' 2. Mo

Re: Curious routing case

Hi Zoran, Thank you for such detailed response. This looks very promising. i will need to get my head around the aggregator pattern. For this week I will be busy with other tasks, but I will get back to it as soon as I can to see if I can get Camel work for the use case. Cheerio Artur On Mon, Ap

Re: Curious routing case

Hi Artur, I was thinking that the order of the messages would be important as you need to process them sequentially. So I think you could use the dynamic message routing[1] with aggregator[2], something like: from("aws-sqs:...") .process("#preProcess") .toD("direct:${header.ne

Re: Curious routing case

Hey Zoran. I read again the patterns you mentioned. In my use case the order of processing within a group doesn't matter as long as two messages from the same group are never processed in parallel. So i guess resenquencer is out of the picture unless I didn't get the intention. So what we are lef

Re: Curious routing case

Hi Artur, have a look at Camel EIP page[1], what you describe sounds to me like Resequencer and Content based router patterns, zoran [1] https://camel.apache.org/eip.html On Fri, Mar 31, 2017 at 5:08 PM, Artur Jablonski wrote: > Hello. > > I wonder if someone could push me in the right directio

Re: Curious routing case

Hmmm. I am getting messages from Amazon sqs and can't change it. Let's say I want to see if I can do it in Camel without putting another messaging system in between. Interesting feature of amq though. Thx! On 31 Mar 2017 5:15 p.m., "Quinn Stevenson" wrote: I’d probably use ActiveMQ Message Grou

Re: Curious routing case

I’d probably use ActiveMQ Message Groups for this http://activemq.apache.org/message-groups.html > On Mar 31, 2017, at 9:08 AM, Artur Jablonski wrote: > > Hello. > > I wonder if someone could push me in the right direction trying to express > q

Curious routing case

Hello. I wonder if someone could push me in the right direction trying to express quite curious case in Camel route. Imagine there's a stream of messages some of which can be processed in parallel and some have to be processed serially. You can group the messages that require serial processing to