On Fri, Jun 29, 2012 at 12:32 AM, David Capwell <dcapw...@gmail.com> wrote: > Playing with the delay fields defined in file2 seems to improve download > speeds. Now that this is working better I want to be able to skip any file > that has already been processed (should persist on restart). It looked to > me that I should be able to set this up by using the filter option, but I > am having a hard time figuring out how to use this option without using > spring. >
Yeah you can either use the filter or the idempotent consumer. The latter is the EIP pattern that fits this problem. The file/ftp component has baked in of both. And the latter has plugins for various persistent storages. The code below seems okay, if you configure the endpoint from the uri, then you need to refer to your bean that implements the filter using the # notation, as you do. Then you need to enlist that bean in the registry. In Spring XML that happens automatic when you do a <bean>. When you use pure Java, you need to pass in a registry implementation when you create the camel context, as you do with the code below using the SimpleRegistry. Then its just a matter of putting the bean in the simple registry, as shown in the code. The unit test of the camel-ftp component has plenty of examples as well https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/ > here is the code I am using (might not compile cause its copy/paste from > diff places): > > SimpleRegistry registry = new SimpleRegistry(); > registry.put("fileExistsFilter", new CopyFilter()); > > CamelContext context = new DefaultCamelContext(registry); > context.addRoutes(new RouteBuilder() { > @Override > public void configure() throws Exception { > from("sftp://example.com/path/to/file?password="; + pass + > "&binary=true&delay="+delay+ "&initialDelay=" + delay + > "&filter=#fileExistsFilter") > .to("file://data/ftp?fileExist=Ignore"); > > } > }); > context.start(); > > ... > public static class CopyFilter implements GenericFileFilter { > > @Override > public boolean accept(GenericFile genericFile) { > // check to see if file exists locally > File localFile = new File("data/ftp", genericFile.getFileName()); > if(localFile.exists()) { > // check to make sure the sizes match > if(localFile.length() == genericFile.getFileLength()) { > // file has already been copied, ignore > return false; > } > } > return true; > } > } > > > Thanks for your time reading this email > > On Thu, Jun 28, 2012 at 6:07 AM, David Capwell <dcapw...@gmail.com> wrote: > >> I'll take a look at file2 to see if that helps. >> >> My goal is to push the data from FTP to s3. Copying locally so I know >> what has been processed, >> On Jun 28, 2012 12:48 AM, "Claus Ibsen" <claus.ib...@gmail.com> wrote: >> >>> Hi >>> >>> As mentioned on the ftp2 page >>> http://camel.apache.org/ftp2 >>> >>> See the file2 page for more options as the ftp component inherits these >>> options >>> http://camel.apache.org/file2.html >>> >>> The delay and consumer.delay option is the same. delay is just >>> shorthand for consumer.delay. >>> >>> And no you cannot avoid the download if you later want to upload it >>> someplace else, and the file exists. >>> The ftp consumer don't know about this. >>> >>> A tricky improvement could be to add a new option to only download the >>> file on demand, but that is a bit tricky >>> as you would need a live connection and the ftp client to still be >>> around and connected. >>> >>> >>> What are you trying to do? To sync files between 2 ftp servers? >>> >>> >>> >>> On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dcapw...@gmail.com> >>> wrote: >>> > I am trying to use sftp (camel 2.9.2) and notice that every download is >>> > spread out to one every 10 seconds (files are around 10kb). Based off >>> the >>> > examples in http://camel.apache.org/ftp2.html it looks like there are >>> delay >>> > options (they are not listened under options, but used in example) but >>> when >>> > I use them nothing changes. How can I have the client pull files >>> faster? >>> > >>> > Another question, if I am trying to write the ftp files locally, is >>> there >>> > any way to avoid the file download if the local file is around? >>> > I append fileExist=Ignore in the to() but it seems that the files still >>> > get written again. >>> > >>> > Here is the route I am using: >>> > >>> > public void ftpToLocal() throws Exception { >>> > runRoutes(new RouteBuilder() { >>> > @Override >>> > public void configure() throws Exception { >>> > Registry registry = getContext().getRegistry(); >>> > String user = (String) registry.lookup("ftpUser"); >>> > String pass = (String) registry.lookup("ftpPass"); >>> > >>> > // we use a delay of 60 minutes (eg. once pr. hour we poll the >>> FTP >>> > server >>> > // long delay = 60 * 60 * 1000L; >>> > // long delay = TimeUnit.SECONDS.toMillis(1); >>> > long delay = 200; >>> > >>> > from("sftp://"+user+"@example.com/path/to/dir?password="; + pass >>> + >>> > "&binary=true&delay="+delay+"&consumer.delay=" + delay) >>> > .to("file://data/ftp?fileExist=Ignore"); >>> > } >>> > }); >>> > } >>> > >>> > >>> > Thanks for your time reading this email. >>> >>> >>> >>> -- >>> Claus Ibsen >>> ----------------- >>> FuseSource >>> Email: cib...@fusesource.com >>> Web: http://fusesource.com >>> Twitter: davsclaus, fusenews >>> Blog: http://davsclaus.com >>> Author of Camel in Action: http://www.manning.com/ibsen >>> >> -- Claus Ibsen ----------------- FuseSource Email: cib...@fusesource.com Web: http://fusesource.com Twitter: davsclaus, fusenews Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen
