I think one more important downside point of using user threads to dispatch
messages from a JMS provider is that you loose guaranteed order of delivery
and the onus of sorting them and guaranteeing message delivery to recipients
is now put onto application code.
I think you would only want to do that in a case where message loss does not
break the application logic and message throughput is more important than
getting all of the produced messages in order. But once you go there why not
use non durable Topics? This is exactly the use case they were designed for.
For example a News Feed application. Each user thread would receive a
message on the topic and deliver it to the destination in parallel.
On Wed, Aug 10, 2011 at 5:48 AM, Claus Ibsen <claus.ib...@gmail.com> wrote:
> On Tue, Aug 9, 2011 at 6:20 PM, Christian Schneider
> <ch...@die-schneider.net> wrote:
> > Hi Claus,
> >
> > without changing anything in camel wouldn´t a simple change in the route
> > also work?
> >
> > from("activemq:queue:inbox?synchronous=true")
> > .threads()
> > .to("jetty:http://0.0.0.0:9432/myapp";)
> > .to("log:result?groupSize=10", "mock:result");
> >
> > So I ask myself if we really need async support in the jms consumer if we
> > can support the scenario using a well known dsl element.
> >
>
> Using threads is not exactly the same. But its is related.
>
> Without threads, you only have the thread of the JMS consumer. And the
> JMS consumer thread will run at a pace that the system can handle. It
> will now consume messages from the JMS broker, faster than Camel can
> handle it. For example if the other pieces in the routes are
> synchronous, then it will keep being synchronous. Its only because
> Jetty supports asynchronous, we are able to scale so high using JMS +
> Jetty.
>
> Here is a link with the components that native support async routing engine
> http://camel.apache.org/asynchronous-routing-engine.html
>
> If you use threads DSL, then you introduce a thread pool, with task
> queue layer. So now you have more active threads in the system. The
> threads DSL have a task queue, so the JMS consumer would be able to
> run faster, then Camel can handle it. By this I mean the JMS consumer
> can consume JMS messages from the JMS broker so fast, and hand over
> the messages to the threads DSL, and have the message stored in the
> thread pool task queue. And it can do this faster than the worker
> threads from the thread pool, so messages will pile up. To remedy this
> you would have to throttle the JMS consumer, or add more worker
> threads, if that is possible. Also you have the complexity what to do
> if the task queue has an upper limit, should you reject the task,
> wait, discard existing tasks, or steal the consumer thread, and have
> it execute as it was a worker thread etc. Also the JMS message would
> be acknowledged and thus consumed from the JMS broker point of view.
> But they are stored in memory in the threads DSL task queue. So in
> case of a crash, or a shutdown, you would have to wait for all those
> tasks to complete. All together you have a more complicated situation.
>
> You can of course customize the threads DSL to have a lower task
> queue. In fact you would be able to have no task queue at all (size of
> 0) but then you have the overhead of handing over the message from the
> JMS consumer to a thread from the threads DSL etc.
>
> Regardless what the current logic in the JmsConsumer is fully
> synchronous so even if you use threads, the JmsConsumer would block.
> So we would have to improve the source code to support this.
>
> And in the future, eg possible with Camel 3.0+ we will support
> asynchronous transactions as well, then this becomes even more
> interesting.
>
>
>
> > Christian
> >
> >
> > Am 09.08.2011 14:25, schrieb Claus Ibsen:
> >>
> >> Hi
> >>
> >> See ticket
> >> https://issues.apache.org/jira/browse/CAMEL-3632
> >>
> >> Take a look at the following routes
> >>
> >> from("activemq:queue:inbox?synchronous=true")
> >> .to("jetty:http://0.0.0.0:9432/myapp";)
> >> .to("log:result?groupSize=10", "mock:result");
> >>
> >> from("jetty:http://0.0.0.0:9432/myapp";)
> >> .delay(100)
> >> .transform(body().prepend("Bye "));
> >>
> >>
> >> Its a fairly simple flow JMS -> JETTY -> MOCK
> >>
> >> The JMS consumer will by default be in synchronous mode. Ticket
> >> CAMEL-3632 is to optimize this by supporting asynchronous routing if
> >> the JMS consumer is *not* transactional, and *not* sending back a
> >> reply message.
> >>
> >> So the scenarios is for high performance one way messaging.
> >>
> >> The 2nd route above simulate a little processing time, which takes 100
> >> millis.
> >>
> >> If we run a test where we send 1000 messages to the queue. How long
> >> time would it take to process those messages synchronously? Of course
> >> roughly 1000 * 0.1 sec = 100 sec. On my laptop it takes:
> >> Took: 1 minute (105393 millis)
> >>
> >> Now imagine we set synchronous=false on the JMS consumer so it runs
> >> asynchronously. How fast would it then go? Well lets try
> >> Took: 10.742 seconds (10742 millis)
> >>
> >> That is a lot fast, well of couse its about 10x faster, since the JMS
> >> consumer is not blocked waiting for the JETTY reply.
> >> However the messages is now not processed in sequence, that is the cost.
> >>
> >> To note in both scenarios this was done with 1 JMS consumer thread. I
> >> did not enable concurrentConsumers. So with the async routing engine
> >> we could scale and utilize the CPU resources better.
> >>
> >> This seems like a great addition. And if we add this, we should add
> >> some documentation with the pros/cons for using this.
> >>
> >> My question is what would a sensitive default setting be? Should we
> >> let the JMS consumer be kept as synchronous by default, to keep it
> >> fully backwards compatible. Then end users must manually set
> >> synchronous=false on the JMS endpoint to enable that.
> >>
> >> Any thoughts?
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> > --
> > Christian Schneider
> > http://www.liquid-reality.de
> >
> > Open Source Architect
> > http://www.talend.com
> >
> >
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cib...@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
