Slow startup of routes
Hi, I am using camel to implement dispatcher EIP. There are thousands of messages in a queue which needs to be delivered at different URLs. Each message has its own delivery URL and delivery protocol (ftp,email,http etc). The way we are implementing in Camel is - Boot a single camel context, the context is disabled for JMX and the loadStatisticsEnabled is set to false on the ManagementStrategy. As mentioned in a jira issue, addressed in 2.11.0 version, for disabling the background management thread creation. - For each message a route is being constructed , the message is being pushed to the route for delivery. - After the message is processed route is shutdown and removed from context. We did a small perf test by having 200 threads of dispatcher component, each sharing the same context. Observed that the time to start a route increases upto a maximum of 60 seconds while the time to process is in milliseconds. Issue CAMEL-5675 mentions that this has been fixed but we are still observing significant time being taken in starting up routes. Pls suggest, what should we do to solve this. The route that we are creating for http is from("direct:"+dispatchItem.getID()).toF("%s?httpClient.soTimeout=%s&disableStreamCache=true", dispatchItem.getEndPointURL(),timeOutInMillis); Each dispatchItem has a unique ID. Thanks, Hussain -- View this message in context: http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Slow startup of routes
Hi, It looks like Camel takes lots of time to create a new Http endpoint, maybe you need to consider to do some caching work for it. BTW, whyou you need to se the soTimeout per endpoint. -- Willem Jiang Red Hat, Inc. FuseSource is now part of Red Hat Web: http://www.fusesource.com | http://www.redhat.com Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/) (English) http://jnn.iteye.com (http://jnn.javaeye.com/) (Chinese) Twitter: willemjiang Weibo: 姜宁willem On Sunday, May 12, 2013 at 3:10 PM, hussainpirosha wrote: > Hi, > > I am using camel to implement dispatcher EIP. There are thousands of > messages in a queue which needs to be delivered at different URLs. Each > message has its own delivery URL and delivery protocol (ftp,email,http etc). > > The way we are implementing in Camel is > - Boot a single camel context, the context is disabled for JMX and the > loadStatisticsEnabled is set to false on the ManagementStrategy. As > mentioned in a jira issue, addressed in 2.11.0 version, for disabling the > background management thread creation. > - For each message a route is being constructed , the message is being > pushed to the route for delivery. > - After the message is processed route is shutdown and removed from context. > > We did a small perf test by having 200 threads of dispatcher component, each > sharing the same context. > Observed that the time to start a route increases upto a maximum of 60 > seconds while the time to process is in milliseconds. > > Issue CAMEL-5675 mentions that this has been fixed but we are still > observing significant time being taken in starting up routes. > > Pls suggest, what should we do to solve this. > > The route that we are creating for http is > > from("direct:"+dispatchItem.getID()).toF("%s?httpClient.soTimeout=%s&disableStreamCache=true", > dispatchItem.getEndPointURL(),timeOutInMillis); > > Each dispatchItem has a unique ID. > > Thanks, > Hussain > > > > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html > Sent from the Camel - Users mailing list archive at Nabble.com > (http://Nabble.com).
Re: Slow startup of routes
Thanks Willem, I am not clear how can we use caching for component creation, please elaborate on it. Need to set the soTimeout per endpoint since each dispatch message contain its endpoint and timeOut settings. Thanks, Hussain -- View this message in context: http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356p5732460.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Slow startup of routes
Hi Your design is probably not good when you need to create a new route to process each message. Instead reuse the route(s) and use the dynamic EIPs if you need to process the messages a bit differently. http://camel.apache.org/how-do-i-use-dynamic-uri-in-to.html On Sun, May 12, 2013 at 9:10 AM, hussainpirosha wrote: > Hi, > > I am using camel to implement dispatcher EIP. There are thousands of > messages in a queue which needs to be delivered at different URLs. Each > message has its own delivery URL and delivery protocol (ftp,email,http etc). > > The way we are implementing in Camel is > - Boot a single camel context, the context is disabled for JMX and the > loadStatisticsEnabled is set to false on the ManagementStrategy. As > mentioned in a jira issue, addressed in 2.11.0 version, for disabling the > background management thread creation. > - For each message a route is being constructed , the message is being > pushed to the route for delivery. > - After the message is processed route is shutdown and removed from context. > > We did a small perf test by having 200 threads of dispatcher component, each > sharing the same context. > Observed that the time to start a route increases upto a maximum of 60 > seconds while the time to process is in milliseconds. > > Issue CAMEL-5675 mentions that this has been fixed but we are still > observing significant time being taken in starting up routes. > > Pls suggest, what should we do to solve this. > > The route that we are creating for http is > > from("direct:"+dispatchItem.getID()).toF("%s?httpClient.soTimeout=%s&disableStreamCache=true", > dispatchItem.getEndPointURL(),timeOutInMillis); > > Each dispatchItem has a unique ID. > > Thanks, > Hussain > > > > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html > Sent from the Camel - Users mailing list archive at Nabble.com. -- Claus Ibsen - www.camelone.org: The open source integration conference. Red Hat, Inc. FuseSource is now part of Red Hat Email: cib...@redhat.com Web: http://fusesource.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen
Re: Slow startup of routes
Thanks Claus for pointing out the mistake. Changed the design to have a single route per protocol and now using the dynamic EIP pattern to pass in the endpoint url and other parameters. The new route looks like : from("direct:http") .recipientList(simple("http:${header.url}? httpClient.soTimeout=${header.timeout}&disableStreamCache=true")); Thanks, Hussain -- View this message in context: http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356p5732473.html Sent from the Camel - Users mailing list archive at Nabble.com.