I am developing an application that downloads large files from a
remote SFTP server over a bandwidth limited network. Download speeds
are bottle-necked, making it highly desirable to be able to download
multiple files concurrently. From what I've gathered, this is not a
feature supported out of the box by the standard SFTP component.
A solution I engineered to work around this was to use a route defined
something like as follows:
from( "sftp:user@host?download=false..." )
.threads( 5 ).maxQueueSize( 0 )
.pollEnrich().simple(
"sftp:user@host?localWorkDirectory=.inbound&include=$simple{file:name}&disconnect=true..."
)
.to( "file:downloads" );
This uses the route's consumer to supply a "list" of files to
download, but delegates the actual downloading to the poll-enrich EIP
across multiple threads. The issue with this approach is that it uses
multiple connections and repeatedly connects and disconnects from the
remote server, which the SFTP server owners dislike.
Is there another existing component or feature that would support the
multiple concurrent downloads use case, or is this a gap in Camel's
feature list?
