Hi
Can you report this issue in JIRA?
The issue is the DEBUG logging will output the body of the file, which
is should not, since it's so big
On Wed, Aug 18, 2010 at 4:13 AM, vcheruvu <v_cher...@hotmail.com> wrote:
>
> Hi,
>
> I have 45MB CSV file. I have created camel route context which simply reads
> from the csv file and routes it to bean for further processing. In doing so,
> i have been getting Java heap space error while Camel tries to process 218K
> lines. Any tips to get around this heap space error other than increasing
> memory? Is there way to read the file in certain sizes or any memory
> optimizing techniques in Camel?
>
> Route config:
>
> <route
> id="file://source/pats?delay=10000&delete=false&exclude=.done&move=.done/${file:name.noext}-${date:now:yyyyMMddHHmmss}&moveFailed=.error/${file:name.noext}-${date:now:yyyyMMddHHmmss}.error&autoCreate=false"/>
> <unmarshal> <csv/> </unmarshal>
> <split>
> <simple>body</simple>
> <to
> uri="bean:saveContractData?method=persistContractData"/>
> </split>
>
> </route>
>
>
> JVM config =
> -Xms1024m -Xmx1024m -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=8
> -XX:NewSize=128m -XX:MaxNewSize=128m -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:PermSize=64m
> -XX:MaxPermSize=64m
>
> 2010-08-18 11:57:53,240 INFO [main] DefaultCamelContext.start:1025 - Apache
> Camel 2.3.0 (CamelContext: camel) started in 735 millis
> 2010-08-18 11:58:58,453 FATAL [Camel Thread 1 -
> file://source/pats?autoCreate=false&delay=10000&delete=false&exclude=.done&move=.done%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D&moveFailed=.error%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D.error]
> ScheduledPollConsumer.run:121 - Consumer
> Consumer[file://source/pats?autoCreate=false&delay=10000&delete=false&exclude=.done&move=.done%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D&moveFailed=.error%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D.error]
> could not poll endpoint:
> file://source/pats?autoCreate=false&delay=10000&delete=false&exclude=.done&move=.done%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D&moveFailed=.error%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D.error
> caused by: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at java.lang.StringBuilder.append(StringBuilder.java:115)
> at java.util.AbstractCollection.toString(AbstractCollection.java:422)
> at java.lang.String.valueOf(String.java:2826)
> at java.lang.StringBuilder.append(StringBuilder.java:115)
> at
> org.apache.camel.component.file.GenericFileMessage.toString(GenericFileMessage.java:58)
> at java.lang.String.valueOf(String.java:2826)
> at java.lang.StringBuilder.append(StringBuilder.java:115)
> at
> org.apache.camel.impl.DefaultExchange.toString(DefaultExchange.java:79)
> at java.lang.String.valueOf(String.java:2826)
> at java.lang.StringBuilder.append(StringBuilder.java:115)
> at
> org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:74)
> at
> org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
> at
> org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:170)
> at
> org.apache.camel.processor.UnitOfWorkProcessor.done(UnitOfWorkProcessor.java:86)
> at
> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:75)
> at
> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:269)
> at
> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:137)
> at
> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:103)
> at
> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:98)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
>
>
>
> Kind regards,
> -Vid-
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Java-heap-space-issue-with-reading-large-CSV-file-tp2638903p2638903.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
