Playing with the delay fields defined in file2 seems to improve download
speeds. Now that this is working better I want to be able to skip any file
that has already been processed (should persist on restart). It looked to
me that I should be able to set this up by using the filter option, but I
am having a hard time figuring out how to use this option without using
spring.
here is the code I am using (might not compile cause its copy/paste from
diff places):
SimpleRegistry registry = new SimpleRegistry();
registry.put("fileExistsFilter", new CopyFilter());
CamelContext context = new DefaultCamelContext(registry);
context.addRoutes(new RouteBuilder() {
@Override
public void configure() throws Exception {
from("sftp://example.com/path/to/file?password="; + pass +
"&binary=true&delay="+delay+ "&initialDelay=" + delay +
"&filter=#fileExistsFilter")
.to("file://data/ftp?fileExist=Ignore");
}
});
context.start();
...
public static class CopyFilter implements GenericFileFilter {
@Override
public boolean accept(GenericFile genericFile) {
// check to see if file exists locally
File localFile = new File("data/ftp", genericFile.getFileName());
if(localFile.exists()) {
// check to make sure the sizes match
if(localFile.length() == genericFile.getFileLength()) {
// file has already been copied, ignore
return false;
}
}
return true;
}
}
Thanks for your time reading this email
On Thu, Jun 28, 2012 at 6:07 AM, David Capwell <[email protected]> wrote:
> I'll take a look at file2 to see if that helps.
>
> My goal is to push the data from FTP to s3. Copying locally so I know
> what has been processed,
> On Jun 28, 2012 12:48 AM, "Claus Ibsen" <[email protected]> wrote:
>
>> Hi
>>
>> As mentioned on the ftp2 page
>> http://camel.apache.org/ftp2
>>
>> See the file2 page for more options as the ftp component inherits these
>> options
>> http://camel.apache.org/file2.html
>>
>> The delay and consumer.delay option is the same. delay is just
>> shorthand for consumer.delay.
>>
>> And no you cannot avoid the download if you later want to upload it
>> someplace else, and the file exists.
>> The ftp consumer don't know about this.
>>
>> A tricky improvement could be to add a new option to only download the
>> file on demand, but that is a bit tricky
>> as you would need a live connection and the ftp client to still be
>> around and connected.
>>
>>
>> What are you trying to do? To sync files between 2 ftp servers?
>>
>>
>>
>> On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <[email protected]>
>> wrote:
>> > I am trying to use sftp (camel 2.9.2) and notice that every download is
>> > spread out to one every 10 seconds (files are around 10kb). Based off
>> the
>> > examples in http://camel.apache.org/ftp2.html it looks like there are
>> delay
>> > options (they are not listened under options, but used in example) but
>> when
>> > I use them nothing changes. How can I have the client pull files
>> faster?
>> >
>> > Another question, if I am trying to write the ftp files locally, is
>> there
>> > any way to avoid the file download if the local file is around?
>> > I append fileExist=Ignore in the to() but it seems that the files still
>> > get written again.
>> >
>> > Here is the route I am using:
>> >
>> > public void ftpToLocal() throws Exception {
>> > runRoutes(new RouteBuilder() {
>> > @Override
>> > public void configure() throws Exception {
>> > Registry registry = getContext().getRegistry();
>> > String user = (String) registry.lookup("ftpUser");
>> > String pass = (String) registry.lookup("ftpPass");
>> >
>> > // we use a delay of 60 minutes (eg. once pr. hour we poll the
>> FTP
>> > server
>> > // long delay = 60 * 60 * 1000L;
>> > // long delay = TimeUnit.SECONDS.toMillis(1);
>> > long delay = 200;
>> >
>> > from("sftp://"+user+"@example.com/path/to/dir?password="; + pass
>> +
>> > "&binary=true&delay="+delay+"&consumer.delay=" + delay)
>> > .to("file://data/ftp?fileExist=Ignore");
>> > }
>> > });
>> > }
>> >
>> >
>> > Thanks for your time reading this email.
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: [email protected]
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>>
>
