On Thu, Sep 20, 2012 at 12:41 PM, Bilgin Ibryam <bibr...@gmail.com> wrote: > Unfortunately we cannot open more than one connection at time, so it is not > possible to have two consumers reading the two (actually now it is three) > files at the same time. > > IMHO this sounds a bit odd/dangerous way of transferring data between >> parties. >> There is no good way to tell if these 2 files are in sync, or the >> first file has new content, and the 2nd has previous content. >> >> It would be better if the other party could change its strategy to >> upload new files using a name pattern to pair the files, eg >> a sequence number that would match etc. >> >> myfile-123.dat >> myotherfile-123.dat >> >> Or to use a 3rd "done" file to tell you that the 2 files has now been >> properly written. eg deleting the done file first, write the 2 files, >> and save the done file again. >> > > Sequencing would be great, but we will not have it. > > > >> >> >> Can you move the files before you download them? As a move operation >> on a remote FTP server is faster than having to download the 2 files >> in parallel in hopes that you make it before the other party starts to >> override the files. >> >> If you can move the files (or rename) then thats likely a near atomic >> IO operation, and thus would be fast doing 2 file moves. Although you >> would need to send 2 FTP commands. >> > > > No, we cannot move or rename the files, because they are consumed also by > other parties, we don't know about. > > >> >> Camel has a preMove option, but its still based on the >> one-file-at-a-time strategy the ftp consumer uses. >> >> >> If you know the file names in advance, then you can have 2 routes, >> each route picking up the designated file. And if you are allowed to >> delete the file after download, then you could assume if there is a >> new file then its new content since last. >> >> But the 2 sec frequency is a bit fast, which mean you would need to >> download and delete the file < 2s. And have the polling frequency of >> the ftp consumer aligned with the other party. >> > > Can you tell me what is 2 sec frequency, where? >
You mentioned that the files will be overwritten with a 2 second frequency. == quote === The reason for that is there is another process which writes to the ftp folder and replaces old files with new ones every couple of seconds. == quote === > > What would be ideal is, if we able to lock the files we want to read before > we start downloading them. > > For example open the two files, and then start the download. The only > guarantee we have is that: a file will not be overridden while we are > reading it. > And how is that possible? Locking files over FTP is "hard". > Do you thing it is possible to extend the component, so that it opens more > than one file, and then start reading them? > No thats not easy the ftp client dont support multi threading. And as well the file/ftp component is based on polling one file at a time. You have a very special use-case. And its possible better to write your own FTP logic to handle that. > > Bilgin > > >> >> >> >> > Thanks >> > Bilgin >> >> >> >> -- >> Claus Ibsen >> ----------------- >> FuseSource >> Email: cib...@fusesource.com >> Web: http://fusesource.com >> Twitter: davsclaus, fusenews >> Blog: http://davsclaus.com >> Author of Camel in Action: http://www.manning.com/ibsen >> -- Claus Ibsen ----------------- Red Hat, Inc. FuseSource is now part of Red Hat Email: cib...@redhat.com Web: http://fusesource.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen
