Hi Brad,
TOPIC 1:
-----------
"StackOverflowError" - this is what happened with the current Route 2 calling
Route 1 for LARGE # OF PAGES. It was not about the data, because the body is
not getting appended and is always limited to one page (at least in most cases).
JVM/stack-size exploded around about 200 calls to the loop :) But yeah a good
learning of what not to do :) Moved the Loop outside the route completely and
based on Route 2 response, triggered the Route 1 again.
Route 1 to Route 2 linkage was done only for modularization purpose.
TOPIC 2:
----------
Very interesting point you have raised here about using POJO and not
Processors. I actually wanted to know is there some explanation of what you
said in CAMEL IN ACTION book. Just got hand of hard-copy and see if I can
understand that better.
I did use split and bean binding, but not sure if that is the right POJO
approach? I guess I should use @ExchangeProperty
- public List<Message> splitMessage(@Body Object body, Exchange
exchange)
Current use cases which processor is doing:
- set / get exchange headers (multiple)
- set / get camel exchange property (so far single)
- saw annotation ExchangeProperty, but it has String type return value.
Whereas on exchange we are setting Objects.
Regards,
Arpit.
-----Original Message-----
From: Brad Johnson [mailto:brad.john...@mediadriver.com]
Sent: Friday, October 14, 2016 8:54 AM
To: users@camel.apache.org
Subject: Re: Looping in routes using Direct Component - Performance impact?
Have you looked at splitter/tokenizer/streaming for this instead of having
route 1 call route 2 which calls route 1? Interesting route setup as I've
never used routes in a recursive fashion like that. When using direct the
return doesn't happen until the end of the last route in the chain. In
this case it appears that you are continually delaying the return from the
route via this iterative approach. I'd have to check that via testing to
see what's really happening in that case but I've never done it like that
and would assume it would eventually run out of memory for large amounts of
data. As far as I know you can't just erase the history though you could
set the body to null or empty string. But I wouldn't do that.
Just unmarshal your data and run it through the splitter in a streaming
fashion. If you don't need to process these in serial order you could then
drop them on a SEDA queue with multiple consumer threads. You can then also
set the queue size to limit how many documents are coming through at a time.
Also, unless you have a compelling reason don't use Processors as they
couple you to the framework. Just use a POJO and let Camel use reflection
to select the correct method for processing or use annotations. As a
general rule of thumb I use, if you don't know exactly why you need to use
a Processor you don't need to use it.
On Thu, Oct 13, 2016 at 6:23 PM, Goyal, Arpit <arpit.go...@sap.com> wrote:
> I can't compromise my business logic and there is hardly any data; but
> each time camel Route 2 starts Route 1, the previous route processing goes
> onto stack. I only have limited stack after that any operation can fail.
>
> To me below approach seems risky...I need to go async or remove the
> looping out of the route overall.
>
> Regards,
> Arpit.
>
> -----Original Message-----
> From: souciance [mailto:souciance.eqdam.ras...@gmail.com]
> Sent: Thursday, October 13, 2016 3:45 PM
> To: users@camel.apache.org
> Subject: Re: Looping in routes using Direct Component - Performance impact?
>
> I think you should first run your route setup with a few pages, 10s of
> pages, hundreds of pages and then thousands of pages and then do
> performance tests to see where what is eating memory and optimise from
> there.
>
> On Fri, Oct 14, 2016 at 12:31 AM, Goyal, Arpit [via Camel] <
> ml-node+s465427n5788744...@n5.nabble.com> wrote:
>
> > Hi,
> >
> > We currently are on 2.16.3 version and can't use the loop function (part
> > of 2.17). So we have the following routes based on which we create sort
> of
> > loop with switch case. Now we are worried about the performance (when we
> > see stacktrace of camel processing)
> >
> > Route 1: from("direct:s1).to("read one page of data").process("process
> > one page").to("direct:s2");
> >
> > Route 2: from("direct:s2").toD("go to external source").when("are more
> > pages left").to("direct:s1").otherwise().process("end processing");
> >
> > Now if there are thousands of pages, it is possible that this loop
> creates
> > lot of objects in memory & can create performance issue? Is there a way
> for
> > Camel to be told, before going to route 1 again, forget all the past? Is
> > disabling HISTORY would solve that?
> >
> > Regards,
> > Arpit.
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> > http://camel.465427.n5.nabble.com/Looping-in-routes-using-
> > Direct-Component-Performance-impact-tp5788744.html
> > To start a new topic under Camel - Users, email
> > ml-node+s465427n465428...@n5.nabble.com
> > To unsubscribe from Camel - Users, click here
> > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=
> unsubscribe_by_code&node=465428&code=c291Y2lhbmNlLmVxZGFtLnJhc2h0aU
> BnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=>
> > .
> > NAML
> > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_
> viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> nabble.view.web.template.NodeNamespace&breadcrumbs=
> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.
> com/Looping-in-routes-using-Direct-Component-Performance-
> impact-tp5788744p5788745.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
