I have added idempotent=true and this seems to work the way i wont but doesn't persist. So to make sure I don't download the same file even after reboot the best option is the filter, correct? If so what am I missing from getting my code to work?
thanks for all the help i have been given so far! On Fri, Jun 29, 2012 at 8:45 AM, David Capwell <dcapw...@gmail.com> wrote: > Sorry, I seemed to miss the exception that I am getting: > > Caused by: java.lang.IllegalArgumentException: Could not find a suitable > setter for property: filter as there isn't a setter method with same type: > java.lang.String nor type conversion possible: No type converter available > to convert from type: java.lang.String to the required type: > org.apache.camel.component.file.GenericFileFilter with value > #fileExistsFilter > > Not sure why but it seems that the filter is not getting the param from > the registry but trying to pass the value in as a normal string. Based on > the last comment it sounds like this should be working? > > > On Thu, Jun 28, 2012 at 11:21 PM, Claus Ibsen <claus.ib...@gmail.com>wrote: > >> On Fri, Jun 29, 2012 at 12:32 AM, David Capwell <dcapw...@gmail.com> >> wrote: >> > Playing with the delay fields defined in file2 seems to improve download >> > speeds. Now that this is working better I want to be able to skip any >> file >> > that has already been processed (should persist on restart). It looked >> to >> > me that I should be able to set this up by using the filter option, but >> I >> > am having a hard time figuring out how to use this option without using >> > spring. >> > >> >> Yeah you can either use the filter or the idempotent consumer. The >> latter is the EIP pattern that fits this problem. >> The file/ftp component has baked in of both. And the latter has >> plugins for various persistent storages. >> >> The code below seems okay, if you configure the endpoint from the uri, >> then you need to refer to your >> bean that implements the filter using the # notation, as you do. >> >> Then you need to enlist that bean in the registry. In Spring XML that >> happens automatic when you do a <bean>. >> When you use pure Java, you need to pass in a registry implementation >> when you create the camel context, >> as you do with the code below using the SimpleRegistry. >> >> Then its just a matter of putting the bean in the simple registry, as >> shown in the code. >> >> >> The unit test of the camel-ftp component has plenty of examples as well >> >> https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/ >> >> >> > here is the code I am using (might not compile cause its copy/paste from >> > diff places): >> > >> > SimpleRegistry registry = new SimpleRegistry(); >> > registry.put("fileExistsFilter", new CopyFilter()); >> > >> > CamelContext context = new DefaultCamelContext(registry); >> > context.addRoutes(new RouteBuilder() { >> > @Override >> > public void configure() throws Exception { >> > from("sftp://example.com/path/to/file?password="; + pass + >> > "&binary=true&delay="+delay+ "&initialDelay=" + delay + >> > "&filter=#fileExistsFilter") >> > .to("file://data/ftp?fileExist=Ignore"); >> > >> > } >> > }); >> > context.start(); >> > >> > ... >> > public static class CopyFilter implements GenericFileFilter { >> > >> > @Override >> > public boolean accept(GenericFile genericFile) { >> > // check to see if file exists locally >> > File localFile = new File("data/ftp", genericFile.getFileName()); >> > if(localFile.exists()) { >> > // check to make sure the sizes match >> > if(localFile.length() == genericFile.getFileLength()) { >> > // file has already been copied, ignore >> > return false; >> > } >> > } >> > return true; >> > } >> > } >> > >> > >> > Thanks for your time reading this email >> > >> > On Thu, Jun 28, 2012 at 6:07 AM, David Capwell <dcapw...@gmail.com> >> wrote: >> > >> >> I'll take a look at file2 to see if that helps. >> >> >> >> My goal is to push the data from FTP to s3. Copying locally so I know >> >> what has been processed, >> >> On Jun 28, 2012 12:48 AM, "Claus Ibsen" <claus.ib...@gmail.com> wrote: >> >> >> >>> Hi >> >>> >> >>> As mentioned on the ftp2 page >> >>> http://camel.apache.org/ftp2 >> >>> >> >>> See the file2 page for more options as the ftp component inherits >> these >> >>> options >> >>> http://camel.apache.org/file2.html >> >>> >> >>> The delay and consumer.delay option is the same. delay is just >> >>> shorthand for consumer.delay. >> >>> >> >>> And no you cannot avoid the download if you later want to upload it >> >>> someplace else, and the file exists. >> >>> The ftp consumer don't know about this. >> >>> >> >>> A tricky improvement could be to add a new option to only download the >> >>> file on demand, but that is a bit tricky >> >>> as you would need a live connection and the ftp client to still be >> >>> around and connected. >> >>> >> >>> >> >>> What are you trying to do? To sync files between 2 ftp servers? >> >>> >> >>> >> >>> >> >>> On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dcapw...@gmail.com> >> >>> wrote: >> >>> > I am trying to use sftp (camel 2.9.2) and notice that every >> download is >> >>> > spread out to one every 10 seconds (files are around 10kb). Based >> off >> >>> the >> >>> > examples in http://camel.apache.org/ftp2.html it looks like there >> are >> >>> delay >> >>> > options (they are not listened under options, but used in example) >> but >> >>> when >> >>> > I use them nothing changes. How can I have the client pull files >> >>> faster? >> >>> > >> >>> > Another question, if I am trying to write the ftp files locally, is >> >>> there >> >>> > any way to avoid the file download if the local file is around? >> >>> > I append fileExist=Ignore in the to() but it seems that the files >> still >> >>> > get written again. >> >>> > >> >>> > Here is the route I am using: >> >>> > >> >>> > public void ftpToLocal() throws Exception { >> >>> > runRoutes(new RouteBuilder() { >> >>> > @Override >> >>> > public void configure() throws Exception { >> >>> > Registry registry = getContext().getRegistry(); >> >>> > String user = (String) registry.lookup("ftpUser"); >> >>> > String pass = (String) registry.lookup("ftpPass"); >> >>> > >> >>> > // we use a delay of 60 minutes (eg. once pr. hour we poll >> the >> >>> FTP >> >>> > server >> >>> > // long delay = 60 * 60 * 1000L; >> >>> > // long delay = TimeUnit.SECONDS.toMillis(1); >> >>> > long delay = 200; >> >>> > >> >>> > from("sftp://"+user+"@example.com/path/to/dir?password="; + >> pass >> >>> + >> >>> > "&binary=true&delay="+delay+"&consumer.delay=" + delay) >> >>> > .to("file://data/ftp?fileExist=Ignore"); >> >>> > } >> >>> > }); >> >>> > } >> >>> > >> >>> > >> >>> > Thanks for your time reading this email. >> >>> >> >>> >> >>> >> >>> -- >> >>> Claus Ibsen >> >>> ----------------- >> >>> FuseSource >> >>> Email: cib...@fusesource.com >> >>> Web: http://fusesource.com >> >>> Twitter: davsclaus, fusenews >> >>> Blog: http://davsclaus.com >> >>> Author of Camel in Action: http://www.manning.com/ibsen >> >>> >> >> >> >> >> >> -- >> Claus Ibsen >> ----------------- >> FuseSource >> Email: cib...@fusesource.com >> Web: http://fusesource.com >> Twitter: davsclaus, fusenews >> Blog: http://davsclaus.com >> Author of Camel in Action: http://www.manning.com/ibsen >> > >
