On Thu, Sep 20, 2012 at 12:55 PM, Bilgin Ibryam <bibr...@gmail.com> wrote: > We don't know the exact frequency yet, but it is definitely more than 2 > seconds, probably up to minute or two, not sure for now. > > hhhmmm, I see it is hard to lock/open more than one file over ftp. > > Then thinking about some tricks, I haven't check it yet, but if ftp > consumer supports the LastModified header as the file component, may be I > can compare this header in the aggregator for the two files, and if they > are not same, ignore the files from this read as they are not in synch, and > try to read it again. WDYT? >
That can be tricky. You would need to make sure the ftp server / ftp client can read the modification timestamp with seconds. I have seen the ftp client report back the timestamp with only precision to the minute, having 00 in seconds always. So give it a bit of test. I am not sure if the output parser of the ftp client needs to be adjusted (eg the one that parses the file name, date, timestamp etc.). The ftp clients have some defaults for various common layouts. And if that dont fit you can use a custom. Also the problem is that if the 2 files are written in about the same timeframe. then you would need to cater for some slack. So maybe have +/- 10 sec or so in different, to match as same file. And there is nothing in the content of the file itself that can tell if they would match? For example a header with some ID etc. Or maybe there is a timstamp as well that is the same for the 2 files, eg report generated timestamp etc. > Thanks > Bilgin > >> >> And how is that possible? Locking files over FTP is "hard". >> >> >> > Do you thing it is possible to extend the component, so that it opens >> more >> > than one file, and then start reading them? >> > >> >> No thats not easy the ftp client dont support multi threading. >> >> And as well the file/ftp component is based on polling one file at a time. >> >> You have a very special use-case. And its possible better to write >> your own FTP logic to handle that. >> >> >> > >> > Bilgin >> > >> > >> >> >> >> >> >> >> >> > Thanks >> >> > Bilgin >> >> >> >> >> >> >> >> -- >> >> Claus Ibsen >> >> ----------------- >> >> FuseSource >> >> Email: cib...@fusesource.com >> >> Web: http://fusesource.com >> >> Twitter: davsclaus, fusenews >> >> Blog: http://davsclaus.com >> >> Author of Camel in Action: http://www.manning.com/ibsen >> >> >> >> >> >> -- >> Claus Ibsen >> ----------------- >> Red Hat, Inc. >> FuseSource is now part of Red Hat >> Email: cib...@redhat.com >> Web: http://fusesource.com >> Twitter: davsclaus >> Blog: http://davsclaus.com >> Author of Camel in Action: http://www.manning.com/ibsen >> -- Claus Ibsen ----------------- Red Hat, Inc. FuseSource is now part of Red Hat Email: cib...@redhat.com Web: http://fusesource.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen
