Hi, I am using Camel 2.7.0 and trying to upload large files using the SFTP
producer. However, past a certain size the transfers fail without any real
informative errors in the log. Interestingly enough, it seems I start to
get failures as soon as the file exceeds my java -Xms heap setting. For
example, my base heap setting is 256m, and I can send a file below that size
without errors, but over that size it fails.
Sounds suspiciously like the whole file is being read into java heap memory,
no? I thought Camel was supposed to avoid this wherever possible, but for
some reason it does not seem to be working for me in this case.
My route is a simple polling file consumer to sftp producer:
Looking at the SFTP component code I can see that the argument to the JSCH
library put() command is an InputStream, and indeed Camel tries to convert
the File body of the exchange to an InputStream to pass to the JSCH lib, but
the conversion fails when the file is too big. Below is some trace ouput
that shows the successful type conversion and the failed one:
#1 File is small enough:
[java] 2011-08-31 23:43:05,863 TRACE
org.apache.camel.component.file.remote.SftpOperations [Camel (camelContext)
thread #0 - file://data/etl-out/, doStoreFile:654]
doStoreFile(SQLEXPRWT_x64_ENU.exe.tmp)
[java] 2011-08-31 22:38:51,957 TRACE
org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext)
thread #0 - file://data/etl-out/, doConvertTo:152] Converting
org.apache.camel.component.file.GenericFile -> java.io.InputStream with
value: GenericFile[SQLEXPRWT_x64_ENU.exe]
[java] 2011-08-31 22:38:51,957 TRACE
org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext)
thread #0 - file://data/etl-out/, doConvertTo:180] Using converter:
StaticMethodTypeConverter: public static java.io.InputStream
org.apache.camel.component.file.GenericFileConverter.genericFileToInputStream(org.apache.camel.component.file.GenericFile,org.apache.camel.Exchange)
throws java.io.IOException to convert [class
org.apache.camel.component.file.GenericFile=>class java.io.InputStream]
[java] 2011-08-31 22:38:51,959 TRACE
org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext)
thread #0 - file://data/etl-out/, doConvertTo:152] Converting java.io.File
-> byte[] with value: data\etl-out\SQLEXPRWT_x64_ENU.exe
[java] 2011-08-31 22:38:51,959 TRACE
org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext)
thread #0 - file://data/etl-out/, doConvertTo:180] Using converter:
StaticMethodTypeConverter: public static byte[]
org.apache.camel.converter.IOConverter.toByteArray(java.io.File) throws
java.io.IOException to convert [class java.io.File=>class [B]
[java] 2011-08-31 22:38:53,520 TRACE
org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext)
thread #0 - file://data/etl-out/, doConvertTo:152] Converting byte[] ->
java.io.InputStream with value: [B@1ebf4ff
[java] 2011-08-31 22:38:53,520 TRACE
org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext)
thread #0 - file://data/etl-out/, doConvertTo:180] Using converter:
StaticMethodTypeConverter: public static java.io.InputStream
org.apache.camel.converter.IOConverter.toInputStream(byte[]) to convert
[class [B=>class java.io.InputStream]
#2 File is too big - subsequent error:
[java] 2011-08-31 23:43:05,863 TRACE
org.apache.camel.component.file.remote.SftpOperations [Camel (camelContext)
thread #0 - file://data/etl-out/ ,doStoreFile:654]
doStoreFile(SQLServer2008SP1-KB968369-x64-ENU.exe.tmp)
[java] 2011-08-31 23:43:05,866 TRACE
org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext)
thread #0 - file://data/etl-out/, doConvertTo:152] Converting
org.apache.camel.component.file.GenericFile -> java.io.InputStream with
value: GenericFile[SQLServer2008SP1-KB968369-x64-ENU.exe]
[java] 2011-08-31 23:43:05,866 TRACE
org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext)
thread #0 - file://data/etl-out/, doConvertTo:180] Using converter:
StaticMethodTypeConverter: public static java.io.InputStream
org.apache.camel.component.file.GenericFileConverter.genericFileToInputStream(org.apache.camel.component.file.GenericFile,org.apache.camel.Exchange)
throws java.io.IOException to convert [class
org.apache.camel.component.file.GenericFile=>class java.io.InputStream]
[java] 2011-08-31 23:43:05,867 TRACE
org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext)
thread #0 - file://data/etl-out/, doConvertTo:152] Converting java.io.File
-> byte[] with value: data\etl-out\SQLServer2008SP1-KB968369-x64-ENU.exe
[java] 2011-08-31 23:43:05,867 TRACE
org.apache.camel.impl.converter.DefaultTypeConverter [Camel (camelContext)
thread #0 - file://data/etl-out/, doConvertTo:180] Using converter:
StaticMethodTypeConverter: public static byte[]
org.apache.camel.converter.IOConverter.toByteArray(java.io.File) throws
java.io.IOException to convert [class java.io.File=>class [B]
...and sometime later...
[java] 2011-08-31 23:43:07,433 DEBUG
org.apache.camel.component.file.GenericFileOnCompletion [Camel
(camelContext) thread #0 - file://data/etl-out/,log:227] Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException -
Cannot store file:
/home/test/data/SQLServer2008SP1-KB968369-x64-ENU.exe.tmp]
[java]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
store file: /home/test/data/SQLServer2008SP1-KB968369-x64-ENU.exe.tmp
[java] at
org.apache.camel.component.file.remote.SftpOperations.doStoreFile(SftpOperations.java:684)[camel-ftp-2.7.0.jar:2.7.0]
[java] at
org.apache.camel.component.file.remote.SftpOperations.storeFile(SftpOperations.java:641)[camel-ftp-2.7.0.jar:2.7.0]
[java] at
org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:269)[camel-core-2.7.0.jar:2.7.0]
[java] at
org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:163)[camel-core-2.7.0.jar:2.7.0]
[java] at
org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:50)[camel-ftp-2.7.0.jar:2.7.0]
[java] at
org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50)[camel-core-2.7.0.jar:2.7.0]
My question is: why does the file get converted first to a byte array before
then being converted to an InputStream? This causes IOConverter to invoke
toBytes() which duplicates the entire stream. Shouldn't you be able to pass
a BufferedInputStream wrapper of the FileInputStream directly to JSCH put()
function? Is this the intended behavior or have I misconfigured something?
Ultimately I have a need to send a 5GB plus file to an SFTP destination --
is it feasible that I should be able to use the SFTP producer for this?
Thanks for reading,
Mike
--
View this message in context:
http://camel.465427.n5.nabble.com/Errors-with-SFTP-producer-and-large-files-tp4757404p4757404.html
Sent from the Camel - Users mailing list archive at Nabble.com.
