On Wed, Aug 18, 2010 at 12:09 PM, Claus Ibsen <claus.ib...@gmail.com> wrote:
> Hi
>
> Can you report this issue in JIRA?
>
> The issue is the DEBUG logging will output the body of the file, which
> is should not, since it's so big
>
btw the workaround is to not use DEBUG level logging on the
org.apache.camel.component.file package.
>
> On Wed, Aug 18, 2010 at 4:13 AM, vcheruvu <v_cher...@hotmail.com> wrote:
>>
>> Hi,
>>
>> I have 45MB CSV file. I have created camel route context which simply reads
>> from the csv file and routes it to bean for further processing. In doing so,
>> i have been getting Java heap space error while Camel tries to process 218K
>> lines. Any tips to get around this heap space error other than increasing
>> memory? Is there way to read the file in certain sizes or any memory
>> optimizing techniques in Camel?
>>
>> Route config:
>>
>> <route
>> id="file://source/pats?delay=10000&delete=false&exclude=.done&move=.done/${file:name.noext}-${date:now:yyyyMMddHHmmss}&moveFailed=.error/${file:name.noext}-${date:now:yyyyMMddHHmmss}.error&autoCreate=false"/>
>> <unmarshal> <csv/> </unmarshal>
>> <split>
>> <simple>body</simple>
>> <to
>> uri="bean:saveContractData?method=persistContractData"/>
>> </split>
>>
>> </route>
>>
>>
>> JVM config =
>> -Xms1024m -Xmx1024m -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=8
>> -XX:NewSize=128m -XX:MaxNewSize=128m -XX:+UseParNewGC
>> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:PermSize=64m
>> -XX:MaxPermSize=64m
>>
>> 2010-08-18 11:57:53,240 INFO [main] DefaultCamelContext.start:1025 - Apache
>> Camel 2.3.0 (CamelContext: camel) started in 735 millis
>> 2010-08-18 11:58:58,453 FATAL [Camel Thread 1 -
>> file://source/pats?autoCreate=false&delay=10000&delete=false&exclude=.done&move=.done%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D&moveFailed=.error%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D.error]
>> ScheduledPollConsumer.run:121 - Consumer
>> Consumer[file://source/pats?autoCreate=false&delay=10000&delete=false&exclude=.done&move=.done%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D&moveFailed=.error%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D.error]
>> could not poll endpoint:
>> file://source/pats?autoCreate=false&delay=10000&delete=false&exclude=.done&move=.done%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D&moveFailed=.error%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D.error
>> caused by: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> at java.util.Arrays.copyOf(Arrays.java:2882)
>> at
>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>> at
>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>> at java.lang.StringBuilder.append(StringBuilder.java:119)
>> at java.lang.StringBuilder.append(StringBuilder.java:115)
>> at java.util.AbstractCollection.toString(AbstractCollection.java:422)
>> at java.lang.String.valueOf(String.java:2826)
>> at java.lang.StringBuilder.append(StringBuilder.java:115)
>> at
>> org.apache.camel.component.file.GenericFileMessage.toString(GenericFileMessage.java:58)
>> at java.lang.String.valueOf(String.java:2826)
>> at java.lang.StringBuilder.append(StringBuilder.java:115)
>> at
>> org.apache.camel.impl.DefaultExchange.toString(DefaultExchange.java:79)
>> at java.lang.String.valueOf(String.java:2826)
>> at java.lang.StringBuilder.append(StringBuilder.java:115)
>> at
>> org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:74)
>> at
>> org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
>> at
>> org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:170)
>> at
>> org.apache.camel.processor.UnitOfWorkProcessor.done(UnitOfWorkProcessor.java:86)
>> at
>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:75)
>> at
>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>> at
>> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:269)
>> at
>> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:137)
>> at
>> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:103)
>> at
>> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:98)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>>
>>
>>
>> Kind regards,
>> -Vid-
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/Java-heap-space-issue-with-reading-large-CSV-file-tp2638903p2638903.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
