[
https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979158#action_12979158
]
Claus Ibsen commented on CAMEL-3497:
------------------------------------
I ran a test with 1.000.000 rows in a file
{code}
2011-01-08 18:25:44,216 [read #9 - Split] INFO split
- Received: 1000000 messages so far. Last group took: 50 millis which is:
20,000 messages per second. average: 17,775.566
2011-01-08 18:25:44,217 [main ] INFO SplitterParallelBigFileTest
- Took 57.423 seconds
2011-01-08 18:25:44,218 [://target/split] INFO route1
- Done splitting bigfile.txt
{code}
And the memory usage was at most 33mb at peak.
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"'
> consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
> Fix For: 3.0.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts
> of heap space for big original messages. E.g. 1024m of heap is not enough to
> process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It
> contains a Future<Exchange> object for every token delivered by the
> java.util.Scanner. The list is only cleared (going out of scope) after all
> Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
