Disabling concurrency in camel route
Hi, I've encountered a scenario in a camel route involving (I believe) concurrency: My route has a netty4 tcp consumer, which then uses a multicast to pass the message through a number of xsl transformers which then output to ActiveMQ endpoints. I've observed that when I receive a large XML message, followed almost immediately (approx. 1 millisecond) by a smaller XML message, the smaller message passes through the route before the large message (obviously to do with the cost of the transformation process). The order of the messages is important, so I'd like to control this behaviour. I'm trying to ensure that only one message is processed by the route at any given time, so would the solution be as simple as setting maxPoolSize=1 on the netty4 consumer? I'm having a hard time understanding the threading/concurrency model in camel, so can anyone help me with a good description somewhere of the lifecycle of a message through a camel route? Thank you in advance, Rob -- View this message in context: http://camel.465427.n5.nabble.com/Disabling-concurrency-in-camel-route-tp5786531.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Disabling concurrency in camel route
You should also do "synchronous=true". Otherwise you may experience reordering on any asynchronous producer. Another option is to use http://camel.apache.org/resequencer.html when you want the sequence restored. Best regards, Vitalii Tymchyshyn Ср, 17 серп. 2016 о 09:28 robina пише: > Hi, > > I've encountered a scenario in a camel route involving (I believe) > concurrency: > > My route has a netty4 tcp consumer, which then uses a multicast to pass the > message through a number of xsl transformers which then output to ActiveMQ > endpoints. > > I've observed that when I receive a large XML message, followed almost > immediately (approx. 1 millisecond) by a smaller XML message, the smaller > message passes through the route before the large message (obviously to do > with the cost of the transformation process). The order of the messages is > important, so I'd like to control this behaviour. > > I'm trying to ensure that only one message is processed by the route at any > given time, so would the solution be as simple as setting maxPoolSize=1 on > the netty4 consumer? > > I'm having a hard time understanding the threading/concurrency model in > camel, so can anyone help me with a good description somewhere of the > lifecycle of a message through a camel route? > > Thank you in advance, > Rob > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Disabling-concurrency-in-camel-route-tp5786531.html > Sent from the Camel - Users mailing list archive at Nabble.com. >
Re: Disabling concurrency in camel route
Thanks for the reply Vitalii, Would setting synchronous=true on its own ensure that only one message is processed by the route at any given time? Or is it the combination of the maxPoolSize of 1 and synchronous=true setting? Regards, Rob -- View this message in context: http://camel.465427.n5.nabble.com/Disabling-concurrency-in-camel-route-tp5786531p5786533.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Disabling concurrency in camel route
I believe you can just set .threads(1) -- View this message in context: http://camel.465427.n5.nabble.com/Disabling-concurrency-in-camel-route-tp5786531p5786535.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Disabling concurrency in camel route
You would need a combination. Ср, 17 серп. 2016 09:41 користувач robina пише: > Thanks for the reply Vitalii, > > Would setting synchronous=true on its own ensure that only one message is > processed by the route at any given time? Or is it the combination of the > maxPoolSize of 1 and synchronous=true setting? > > Regards, > Rob > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Disabling-concurrency-in-camel-route-tp5786531p5786533.html > Sent from the Camel - Users mailing list archive at Nabble.com. >
Re: Disabling concurrency in camel route
When I’ve hit situations like this, I’ve used the SEDA component to serialize the processing after the message was received. All of the processing winds up in the SEDA route, and the actual receiving route (in this case, from Netty) just calls the SEDA route. Something like > > On Aug 17, 2016, at 8:47 AM, Vitalii Tymchyshyn wrote: > > You would need a combination. > > Ср, 17 серп. 2016 09:41 користувач robina пише: > >> Thanks for the reply Vitalii, >> >> Would setting synchronous=true on its own ensure that only one message is >> processed by the route at any given time? Or is it the combination of the >> maxPoolSize of 1 and synchronous=true setting? >> >> Regards, >> Rob >> >> >> >> -- >> View this message in context: >> http://camel.465427.n5.nabble.com/Disabling-concurrency-in-camel-route-tp5786531p5786533.html >> Sent from the Camel - Users mailing list archive at Nabble.com. >>
Re: Disabling concurrency in camel route
I'd second what Quinn said. The other nice thing about the SEDA queue is you can directly configure the number of threads that are consuming from it. On Thu, Sep 1, 2016 at 1:50 PM, Quinn Stevenson wrote: > When I’ve hit situations like this, I’ve used the SEDA component to > serialize the processing after the message was received. All of the > processing winds up in the SEDA route, and the actual receiving route (in > this case, from Netty) just calls the SEDA route. Something like > > > > > > > > > > > > > > > > On Aug 17, 2016, at 8:47 AM, Vitalii Tymchyshyn wrote: > > > > You would need a combination. > > > > Ср, 17 серп. 2016 09:41 користувач robina пише: > > > >> Thanks for the reply Vitalii, > >> > >> Would setting synchronous=true on its own ensure that only one message > is > >> processed by the route at any given time? Or is it the combination of > the > >> maxPoolSize of 1 and synchronous=true setting? > >> > >> Regards, > >> Rob > >> > >> > >> > >> -- > >> View this message in context: > >> http://camel.465427.n5.nabble.com/Disabling-concurrency-in- > camel-route-tp5786531p5786533.html > >> Sent from the Camel - Users mailing list archive at Nabble.com. > >> > >
Re: Disabling concurrency in camel route
I’d be careful with configuring the number of consumers on the SEDA queue if you need to preserve order - you’ll likely get stuff processed out-of-order. > On Sep 1, 2016, at 1:00 PM, Brad Johnson wrote: > > I'd second what Quinn said. The other nice thing about the SEDA queue is > you can directly configure the number of threads that are consuming from it. > > On Thu, Sep 1, 2016 at 1:50 PM, Quinn Stevenson > wrote: > >> When I’ve hit situations like this, I’ve used the SEDA component to >> serialize the processing after the message was received. All of the >> processing winds up in the SEDA route, and the actual receiving route (in >> this case, from Netty) just calls the SEDA route. Something like >> >> >> >> >> >> >> >>> >> >> >> >> >> >>> On Aug 17, 2016, at 8:47 AM, Vitalii Tymchyshyn wrote: >>> >>> You would need a combination. >>> >>> Ср, 17 серп. 2016 09:41 користувач robina пише: >>> >>>> Thanks for the reply Vitalii, >>>> >>>> Would setting synchronous=true on its own ensure that only one message >> is >>>> processed by the route at any given time? Or is it the combination of >> the >>>> maxPoolSize of 1 and synchronous=true setting? >>>> >>>> Regards, >>>> Rob >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://camel.465427.n5.nabble.com/Disabling-concurrency-in- >> camel-route-tp5786531p5786533.html >>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>> >> >>
Re: Disabling concurrency in camel route
True enough. I used a resequencer to make sure they went in and out.
${body}
exchangeProperty.CamelSplitIndex
...
in.header.seqnum
On Thu, Sep 1, 2016 at 2:08 PM, Quinn Stevenson wrote:
> I’d be careful with configuring the number of consumers on the SEDA queue
> if you need to preserve order - you’ll likely get stuff processed
> out-of-order.
>
>
> > On Sep 1, 2016, at 1:00 PM, Brad Johnson
> wrote:
> >
> > I'd second what Quinn said. The other nice thing about the SEDA queue is
> > you can directly configure the number of threads that are consuming from
> it.
> >
> > On Thu, Sep 1, 2016 at 1:50 PM, Quinn Stevenson <
> qu...@pronoia-solutions.com
> >> wrote:
> >
> >> When I’ve hit situations like this, I’ve used the SEDA component to
> >> serialize the processing after the message was received. All of the
> >> processing winds up in the SEDA route, and the actual receiving route
> (in
> >> this case, from Netty) just calls the SEDA route. Something like
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>>
> >>
> >>
> >>
> >>
> >>
> >>> On Aug 17, 2016, at 8:47 AM, Vitalii Tymchyshyn wrote:
> >>>
> >>> You would need a combination.
> >>>
> >>> Ср, 17 серп. 2016 09:41 користувач robina пише:
> >>>
> >>>> Thanks for the reply Vitalii,
> >>>>
> >>>> Would setting synchronous=true on its own ensure that only one message
> >> is
> >>>> processed by the route at any given time? Or is it the combination of
> >> the
> >>>> maxPoolSize of 1 and synchronous=true setting?
> >>>>
> >>>> Regards,
> >>>> Rob
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> View this message in context:
> >>>> http://camel.465427.n5.nabble.com/Disabling-concurrency-in-
> >> camel-route-tp5786531p5786533.html
> >>>> Sent from the Camel - Users mailing list archive at Nabble.com.
> >>>>
> >>
> >>
>
>
Re: Disabling concurrency in camel route
True enough. I've had to deal with that before by using an aggregator and
assigning sequential numbers to the records as they go through and the
aggregate the results. Obviously that won't work if the records have to be
processed in the exact order they come in and not just written out in the
same order. I did in the case of processing credit cards where the
transactions could go in any order but they wanted the results written back
out in the same order they came in in the input file. One of the files had
a footer that came in and had to be written out as a footer so the
aggregation was important for that.
${body}
exchangeProperty.CamelSplitIndex
On Thu, Sep 1, 2016 at 2:08 PM, Quinn Stevenson wrote:
> I’d be careful with configuring the number of consumers on the SEDA queue
> if you need to preserve order - you’ll likely get stuff processed
> out-of-order.
>
>
> > On Sep 1, 2016, at 1:00 PM, Brad Johnson
> wrote:
> >
> > I'd second what Quinn said. The other nice thing about the SEDA queue is
> > you can directly configure the number of threads that are consuming from
> it.
> >
> > On Thu, Sep 1, 2016 at 1:50 PM, Quinn Stevenson <
> qu...@pronoia-solutions.com
> >> wrote:
> >
> >> When I’ve hit situations like this, I’ve used the SEDA component to
> >> serialize the processing after the message was received. All of the
> >> processing winds up in the SEDA route, and the actual receiving route
> (in
> >> this case, from Netty) just calls the SEDA route. Something like
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>>
> >>
> >>
> >>
> >>
> >>
> >>> On Aug 17, 2016, at 8:47 AM, Vitalii Tymchyshyn wrote:
> >>>
> >>> You would need a combination.
> >>>
> >>> Ср, 17 серп. 2016 09:41 користувач robina пише:
> >>>
> >>>> Thanks for the reply Vitalii,
> >>>>
> >>>> Would setting synchronous=true on its own ensure that only one message
> >> is
> >>>> processed by the route at any given time? Or is it the combination of
> >> the
> >>>> maxPoolSize of 1 and synchronous=true setting?
> >>>>
> >>>> Regards,
> >>>> Rob
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> View this message in context:
> >>>> http://camel.465427.n5.nabble.com/Disabling-concurrency-in-
> >> camel-route-tp5786531p5786533.html
> >>>> Sent from the Camel - Users mailing list archive at Nabble.com.
> >>>>
> >>
> >>
>
>
