Slow startup of routes
Hi,
I am using camel to implement dispatcher EIP. There are thousands of
messages in a queue which needs to be delivered at different URLs. Each
message has its own delivery URL and delivery protocol (ftp,email,http etc).
The way we are implementing in Camel is
- Boot a single camel context, the context is disabled for JMX and the
loadStatisticsEnabled is set to false on the ManagementStrategy. As
mentioned in a jira issue, addressed in 2.11.0 version, for disabling the
background management thread creation.
- For each message a route is being constructed , the message is being
pushed to the route for delivery.
- After the message is processed route is shutdown and removed from context.
We did a small perf test by having 200 threads of dispatcher component, each
sharing the same context.
Observed that the time to start a route increases upto a maximum of 60
seconds while the time to process is in milliseconds.
Issue CAMEL-5675 mentions that this has been fixed but we are still
observing significant time being taken in starting up routes.
Pls suggest, what should we do to solve this.
The route that we are creating for http is
from("direct:"+dispatchItem.getID()).toF("%s?httpClient.soTimeout=%s&disableStreamCache=true",
dispatchItem.getEndPointURL(),timeOutInMillis);
Each dispatchItem has a unique ID.
Thanks,
Hussain
--
View this message in context:
http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Slow startup of routes
Hi,
It looks like Camel takes lots of time to create a new Http endpoint, maybe you
need to consider to do some caching work for it.
BTW, whyou you need to se the soTimeout per endpoint.
--
Willem Jiang
Red Hat, Inc.
FuseSource is now part of Red Hat
Web: http://www.fusesource.com | http://www.redhat.com
Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/)
(English)
http://jnn.iteye.com (http://jnn.javaeye.com/) (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On Sunday, May 12, 2013 at 3:10 PM, hussainpirosha wrote:
> Hi,
>
> I am using camel to implement dispatcher EIP. There are thousands of
> messages in a queue which needs to be delivered at different URLs. Each
> message has its own delivery URL and delivery protocol (ftp,email,http etc).
>
> The way we are implementing in Camel is
> - Boot a single camel context, the context is disabled for JMX and the
> loadStatisticsEnabled is set to false on the ManagementStrategy. As
> mentioned in a jira issue, addressed in 2.11.0 version, for disabling the
> background management thread creation.
> - For each message a route is being constructed , the message is being
> pushed to the route for delivery.
> - After the message is processed route is shutdown and removed from context.
>
> We did a small perf test by having 200 threads of dispatcher component, each
> sharing the same context.
> Observed that the time to start a route increases upto a maximum of 60
> seconds while the time to process is in milliseconds.
>
> Issue CAMEL-5675 mentions that this has been fixed but we are still
> observing significant time being taken in starting up routes.
>
> Pls suggest, what should we do to solve this.
>
> The route that we are creating for http is
>
> from("direct:"+dispatchItem.getID()).toF("%s?httpClient.soTimeout=%s&disableStreamCache=true",
> dispatchItem.getEndPointURL(),timeOutInMillis);
>
> Each dispatchItem has a unique ID.
>
> Thanks,
> Hussain
>
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html
> Sent from the Camel - Users mailing list archive at Nabble.com
> (http://Nabble.com).
Re: Slow startup of routes
Thanks Willem, I am not clear how can we use caching for component creation, please elaborate on it. Need to set the soTimeout per endpoint since each dispatch message contain its endpoint and timeOut settings. Thanks, Hussain -- View this message in context: http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356p5732460.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Slow startup of routes
Hi
Your design is probably not good when you need to create a new route
to process each message.
Instead reuse the route(s) and use the dynamic EIPs if you need to
process the messages a bit differently.
http://camel.apache.org/how-do-i-use-dynamic-uri-in-to.html
On Sun, May 12, 2013 at 9:10 AM, hussainpirosha
wrote:
> Hi,
>
> I am using camel to implement dispatcher EIP. There are thousands of
> messages in a queue which needs to be delivered at different URLs. Each
> message has its own delivery URL and delivery protocol (ftp,email,http etc).
>
> The way we are implementing in Camel is
> - Boot a single camel context, the context is disabled for JMX and the
> loadStatisticsEnabled is set to false on the ManagementStrategy. As
> mentioned in a jira issue, addressed in 2.11.0 version, for disabling the
> background management thread creation.
> - For each message a route is being constructed , the message is being
> pushed to the route for delivery.
> - After the message is processed route is shutdown and removed from context.
>
> We did a small perf test by having 200 threads of dispatcher component, each
> sharing the same context.
> Observed that the time to start a route increases upto a maximum of 60
> seconds while the time to process is in milliseconds.
>
> Issue CAMEL-5675 mentions that this has been fixed but we are still
> observing significant time being taken in starting up routes.
>
> Pls suggest, what should we do to solve this.
>
> The route that we are creating for http is
>
> from("direct:"+dispatchItem.getID()).toF("%s?httpClient.soTimeout=%s&disableStreamCache=true",
> dispatchItem.getEndPointURL(),timeOutInMillis);
>
> Each dispatchItem has a unique ID.
>
> Thanks,
> Hussain
>
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-
www.camelone.org: The open source integration conference.
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cib...@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Slow startup of routes
Thanks Claus for pointing out the mistake.
Changed the design to have a single route per protocol and now using the
dynamic EIP pattern to pass in the endpoint url and other parameters. The
new route looks like :
from("direct:http")
.recipientList(simple("http:${header.url}?
httpClient.soTimeout=${header.timeout}&disableStreamCache=true"));
Thanks,
Hussain
--
View this message in context:
http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356p5732473.html
Sent from the Camel - Users mailing list archive at Nabble.com.
