[
https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979113#action_12979113
]
Claus Ibsen commented on CAMEL-3497:
------------------------------------
Okay good news. I refactored the logic so Camel now aggregates the parallel
tasks on-the-fly.
This makes a tremendous difference. Now I can split a file into 50.000 sub
messages and process that in 7 sec, using at most 18mb.
Before I would hit an issue at about 25.000-30.000 message and hit OOME with
130mb.
Since the logic is more complex because there is a separate tasks which
aggregates on the fly, while the other task submit new tasks, there is logic to
signal between the two tasks. They kinda need to agree when there are no more
messages to split, and when it has aggregated all of those.
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"'
> consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
> Fix For: 3.0.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts
> of heap space for big original messages. E.g. 1024m of heap is not enough to
> process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It
> contains a Future<Exchange> object for every token delivered by the
> java.util.Scanner. The list is only cleared (going out of scope) after all
> Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
