Hi Use the localWorkDirectory option on the FTP endpoint to stream from FTP directly to files in a local work dir to avoid loading those big files into memory.
On Fri, Jan 29, 2010 at 5:08 PM, Kevin Jackson <foamd...@gmail.com> wrote: > Hi, > > We have some large files which we have to process from an FTP server. > Currently we have to run camel with -Xmx2048m to allow these files to > process without camel failing with an (unreported) OutOfMemory error > (which we picked up with -XX:+HeapDumpOnOutOfMemoryError) > > Reading the documentation points in the direction of not using the > default idempotent consumer[1]. I suppose I will have to use some > kind of JPA or filebased idempotent store - I don't like the idea of > this, so I'm trying to come up with a way of avoiding it. > > We currently have the following Spring route: > > <camel:route> > <camel:from ref="stock-schedule"/> > </camel:route> > > <camel:route id="stock-route-download" startupOrder="0"> > <camel:from ref="stock-ftp"/> > <camel:to ref="stock-file"/> > </camel:route> > > <camel:route id="stock-file-processing-route"> > <camel:from ref="stock-file-consumer"/> > <camel:bean ref="stockUnmarshaller" method="unmarshall"/> > <camel:split streaming="true"> > <camel:method bean="stockTransformer" method="transform"/> > <camel:bean ref="persister" method="persist"/> > </camel:split> > </camel:route> > > In this we are consuming a 100+Mb file in one gulp, then splitting and > streaming the generated objects to the db. If I was to move the > splitter upstream: > > <camel:route id="stock-file-processing-route"> > <camel:from ref="stock-file-consumer"/> > <camel:split streaming="true"> > <camel:bean ref="stockUnmarshaller" method="unmarshall"/> > <camel:method bean="stockTransformer" method="transform"/> > <camel:bean ref="persister" method="persist"/> > </camel:split> > </camel:route> > > Would this reduce the memory consumption? This is just one of many > routes, and I'm concerned about the apparent wastefulness of camel at > the moment (although I'm fairly convinced that it's my config that is > incorrect). > > [1]http://camel.apache.org/idempotent-consumer.html > -- Claus Ibsen Apache Camel Committer Author of Camel in Action: http://www.manning.com/ibsen/ Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/ Twitter: http://twitter.com/davsclaus
