Also, use seda for async processing and you can shutoff some camel stuff like jmx, original message etc for increased performance. But for performance you need to do good tests and make a change and repeat those tests and measure.
On Fri, Oct 14, 2016 at 5:54 PM, Brad Johnson [via Camel] < ml-node+s465427n578881...@n5.nabble.com> wrote: > Have you looked at splitter/tokenizer/streaming for this instead of having > route 1 call route 2 which calls route 1? Interesting route setup as I've > never used routes in a recursive fashion like that. When using direct the > return doesn't happen until the end of the last route in the chain. In > this case it appears that you are continually delaying the return from the > route via this iterative approach. I'd have to check that via testing to > see what's really happening in that case but I've never done it like that > and would assume it would eventually run out of memory for large amounts > of > data. As far as I know you can't just erase the history though you could > set the body to null or empty string. But I wouldn't do that. > > Just unmarshal your data and run it through the splitter in a streaming > fashion. If you don't need to process these in serial order you could > then > drop them on a SEDA queue with multiple consumer threads. You can then > also > set the queue size to limit how many documents are coming through at a > time. > > Also, unless you have a compelling reason don't use Processors as they > couple you to the framework. Just use a POJO and let Camel use reflection > to select the correct method for processing or use annotations. As a > general rule of thumb I use, if you don't know exactly why you need to use > a Processor you don't need to use it. > > > On Thu, Oct 13, 2016 at 6:23 PM, Goyal, Arpit <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=5788811&i=0>> wrote: > > > I can't compromise my business logic and there is hardly any data; but > > each time camel Route 2 starts Route 1, the previous route processing > goes > > onto stack. I only have limited stack after that any operation can fail. > > > > To me below approach seems risky...I need to go async or remove the > > looping out of the route overall. > > > > Regards, > > Arpit. > > > > -----Original Message----- > > From: souciance [mailto:[hidden email] > <http:///user/SendEmail.jtp?type=node&node=5788811&i=1>] > > Sent: Thursday, October 13, 2016 3:45 PM > > To: [hidden email] > <http:///user/SendEmail.jtp?type=node&node=5788811&i=2> > > Subject: Re: Looping in routes using Direct Component - Performance > impact? > > > > I think you should first run your route setup with a few pages, 10s of > > pages, hundreds of pages and then thousands of pages and then do > > performance tests to see where what is eating memory and optimise from > > there. > > > > On Fri, Oct 14, 2016 at 12:31 AM, Goyal, Arpit [via Camel] < > > [hidden email] <http:///user/SendEmail.jtp?type=node&node=5788811&i=3>> > wrote: > > > > > Hi, > > > > > > We currently are on 2.16.3 version and can't use the loop function > (part > > > of 2.17). So we have the following routes based on which we create > sort > > of > > > loop with switch case. Now we are worried about the performance (when > we > > > see stacktrace of camel processing) > > > > > > Route 1: from("direct:s1).to("read one page of > data").process("process > > > one page").to("direct:s2"); > > > > > > Route 2: from("direct:s2").toD("go to external source").when("are > more > > > pages left").to("direct:s1").otherwise().process("end processing"); > > > > > > Now if there are thousands of pages, it is possible that this loop > > creates > > > lot of objects in memory & can create performance issue? Is there a > way > > for > > > Camel to be told, before going to route 1 again, forget all the past? > Is > > > disabling HISTORY would solve that? > > > > > > Regards, > > > Arpit. > > > > > > > > > ------------------------------ > > > If you reply to this email, your message will be added to the > discussion > > > below: > > > http://camel.465427.n5.nabble.com/Looping-in-routes-using- > > > Direct-Component-Performance-impact-tp5788744.html > > > To start a new topic under Camel - Users, email > > > [hidden email] <http:///user/SendEmail.jtp?type=node&node=5788811&i=4> > > > To unsubscribe from Camel - Users, click here > > > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro= > > unsubscribe_by_code&node=465428&code=c291Y2lhbmNlLmVxZGFtLnJhc2h0aU > > BnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=> > > > . > > > NAML > > > <http://camel.465427.n5.nabble.com/template/ > NamlServlet.jtp?macro=macro_ > > viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces. > > > BasicNamespace-nabble.view.web.template.NabbleNamespace- > > nabble.view.web.template.NodeNamespace&breadcrumbs= > > notify_subscribers%21nabble%3Aemail.naml-instant_emails% > > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > > > > > > > > > > > -- > > View this message in context: http://camel.465427.n5.nabble. > > com/Looping-in-routes-using-Direct-Component-Performance- > > impact-tp5788744p5788745.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://camel.465427.n5.nabble.com/Looping-in-routes-using- > Direct-Component-Performance-impact-tp5788744p5788811.html > To start a new topic under Camel - Users, email > ml-node+s465427n465428...@n5.nabble.com > To unsubscribe from Camel - Users, click here > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=c291Y2lhbmNlLmVxZGFtLnJhc2h0aUBnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=> > . > NAML > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://camel.465427.n5.nabble.com/Looping-in-routes-using-Direct-Component-Performance-impact-tp5788744p5788819.html Sent from the Camel - Users mailing list archive at Nabble.com.
