Hi The ftp component has a localWorkDirectory option which you can use to configure a temp directory, which allows the FTP component to stream directly to a java.io.File instead of in memory.
The option is documented in these 2 pages http://camel.apache.org/ftp2 http://camel.apache.org/file2 On Thu, Sep 1, 2011 at 9:39 AM, Mike D <[email protected]> wrote: > Hi, I am using Camel 2.7.0 and trying to upload large files using the SFTP > producer. However, past a certain size the transfers fail without any real > informative errors in the log. Interestingly enough, it seems I start to > get failures as soon as the file exceeds my java -Xms heap setting. For > example, my base heap setting is 256m, and I can send a file below that size > without errors, but over that size it fails. > Sounds suspiciously like the whole file is being read into java heap memory, > no? I thought Camel was supposed to avoid this wherever possible, but for > some reason it does not seem to be working for me in this case. > > My route is a simple polling file consumer to sftp producer: > > > Looking at the SFTP component code I can see that the argument to the JSCH > library put() command is an InputStream, and indeed Camel tries to convert > the File body of the exchange to an InputStream to pass to the JSCH lib, but > the conversion fails when the file is too big. Below is some trace ouput > that shows the successful type conversion and the failed one: > > #1 File is small enough: > > [java] 2011-08-31 23:43:05,863 TRACE > org.apache.camel.component.file.remote.SftpOperations [Camel (camelContext) > thread #0 - file://data/etl-out/, doStoreFile:654] > doStoreFile(SQLEXPRWT_x64_ENU.exe.tmp) > [java] 2011-08-31 22:38:51,957 TRACE > org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext) > thread #0 - file://data/etl-out/, doConvertTo:152] Converting > org.apache.camel.component.file.GenericFile -> java.io.InputStream with > value: GenericFile[SQLEXPRWT_x64_ENU.exe] > [java] 2011-08-31 22:38:51,957 TRACE > org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext) > thread #0 - file://data/etl-out/, doConvertTo:180] Using converter: > StaticMethodTypeConverter: public static java.io.InputStream > org.apache.camel.component.file.GenericFileConverter.genericFileToInputStream(org.apache.camel.component.file.GenericFile,org.apache.camel.Exchange) > throws java.io.IOException to convert [class > org.apache.camel.component.file.GenericFile=>class java.io.InputStream] > [java] 2011-08-31 22:38:51,959 TRACE > org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext) > thread #0 - file://data/etl-out/, doConvertTo:152] Converting java.io.File > -> byte[] with value: data\etl-out\SQLEXPRWT_x64_ENU.exe > [java] 2011-08-31 22:38:51,959 TRACE > org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext) > thread #0 - file://data/etl-out/, doConvertTo:180] Using converter: > StaticMethodTypeConverter: public static byte[] > org.apache.camel.converter.IOConverter.toByteArray(java.io.File) throws > java.io.IOException to convert [class java.io.File=>class [B] > [java] 2011-08-31 22:38:53,520 TRACE > org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext) > thread #0 - file://data/etl-out/, doConvertTo:152] Converting byte[] -> > java.io.InputStream with value: [B@1ebf4ff > [java] 2011-08-31 22:38:53,520 TRACE > org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext) > thread #0 - file://data/etl-out/, doConvertTo:180] Using converter: > StaticMethodTypeConverter: public static java.io.InputStream > org.apache.camel.converter.IOConverter.toInputStream(byte[]) to convert > [class [B=>class java.io.InputStream] > > #2 File is too big - subsequent error: > > [java] 2011-08-31 23:43:05,863 TRACE > org.apache.camel.component.file.remote.SftpOperations [Camel (camelContext) > thread #0 - file://data/etl-out/ ,doStoreFile:654] > doStoreFile(SQLServer2008SP1-KB968369-x64-ENU.exe.tmp) > [java] 2011-08-31 23:43:05,866 TRACE > org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext) > thread #0 - file://data/etl-out/, doConvertTo:152] Converting > org.apache.camel.component.file.GenericFile -> java.io.InputStream with > value: GenericFile[SQLServer2008SP1-KB968369-x64-ENU.exe] > [java] 2011-08-31 23:43:05,866 TRACE > org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext) > thread #0 - file://data/etl-out/, doConvertTo:180] Using converter: > StaticMethodTypeConverter: public static java.io.InputStream > org.apache.camel.component.file.GenericFileConverter.genericFileToInputStream(org.apache.camel.component.file.GenericFile,org.apache.camel.Exchange) > throws java.io.IOException to convert [class > org.apache.camel.component.file.GenericFile=>class java.io.InputStream] > [java] 2011-08-31 23:43:05,867 TRACE > org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext) > thread #0 - file://data/etl-out/, doConvertTo:152] Converting java.io.File > -> byte[] with value: data\etl-out\SQLServer2008SP1-KB968369-x64-ENU.exe > [java] 2011-08-31 23:43:05,867 TRACE > org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext) > thread #0 - file://data/etl-out/, doConvertTo:180] Using converter: > StaticMethodTypeConverter: public static byte[] > org.apache.camel.converter.IOConverter.toByteArray(java.io.File) throws > java.io.IOException to convert [class java.io.File=>class [B] > > ...and sometime later... > > [java] 2011-08-31 23:43:07,433 DEBUG > org.apache.camel.component.file.GenericFileOnCompletion [Camel > (camelContext) thread #0 - file://data/etl-out/,log:227] Caused by: > [org.apache.camel.component.file.GenericFileOperationFailedException - > Cannot store file: > /home/test/data/SQLServer2008SP1-KB968369-x64-ENU.exe.tmp] > [java] > org.apache.camel.component.file.GenericFileOperationFailedException: Cannot > store file: /home/test/data/SQLServer2008SP1-KB968369-x64-ENU.exe.tmp > [java] at > org.apache.camel.component.file.remote.SftpOperations.doStoreFile(SftpOperations.java:684)[camel-ftp-2.7.0.jar:2.7.0] > [java] at > org.apache.camel.component.file.remote.SftpOperations.storeFile(SftpOperations.java:641)[camel-ftp-2.7.0.jar:2.7.0] > [java] at > org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:269)[camel-core-2.7.0.jar:2.7.0] > [java] at > org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:163)[camel-core-2.7.0.jar:2.7.0] > [java] at > org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:50)[camel-ftp-2.7.0.jar:2.7.0] > [java] at > org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50)[camel-core-2.7.0.jar:2.7.0] > > > My question is: why does the file get converted first to a byte array before > then being converted to an InputStream? This causes IOConverter to invoke > toBytes() which duplicates the entire stream. Shouldn't you be able to pass > a BufferedInputStream wrapper of the FileInputStream directly to JSCH put() > function? Is this the intended behavior or have I misconfigured something? > Ultimately I have a need to send a 5GB plus file to an SFTP destination -- > is it feasible that I should be able to use the SFTP producer for this? > > Thanks for reading, > Mike > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Errors-with-SFTP-producer-and-large-files-tp4757404p4757404.html > Sent from the Camel - Users mailing list archive at Nabble.com. > -- Claus Ibsen ----------------- FuseSource Email: [email protected] Web: http://fusesource.com Twitter: davsclaus, fusenews Blog: http://davsclaus.blogspot.com/ Author of Camel in Action: http://www.manning.com/ibsen/
