Hi Yeah the current changed read-lock afair don't keep state between polls. So we could look into one that does that, and do a full scan of all the files, and then do a change detect on all files all together and make up which ones hasn't changed. And that way can react faster than currently.
Now that may require doing more file directory listings to gather all those files and their timestamps / size to see which one has changed, instead of monitoring a single file one by one. Also it may mean that files can be processed out of order, if a file sort isn't must be strictly followed. On Thu, Nov 12, 2015 at 10:05 PM, David Hoffer <dhoff...@gmail.com> wrote: > I'm rather new to Camel, I'm assuming that if the SFTP source gets 900 > files then the configured > GenericFileExclusiveReadLockStrategy#acquireExclusiveReadLock will get > called 900 times (once per file) every polling cycle. > > We too have a custom GenericFileExclusiveReadLockStrategy class that does > not block. What ours does is maintain a map/cache of of the > lastModifiedTime and fileSize for each of the 900 files. The call to > acquireExclusiveReadLock() will return true as soon as neither of those > values changed since the last poll cycle. > > We have our delay/poll cycle set to 60 seconds. I'm not sure why such a > large value but we are fine waiting 2 minutes to get each of the 900 > files...and by the time 2 minutes go by we will likely have more files in > the SFTP source. E.g. if we have a constant delay in receiving each file > of 2-3 minutes that is more than fine. The problem is that on each cycle > it only processes 3-5 files instead of the 900 that should have returned > true from acquireExclusiveReadLock() > > What seems to be happening is that Camel is not calling the > acquireExclusiveReadLock() method for each of the 900 files every 60 > seconds, rather it slows down and either calls just a few of those 900 > files or none. > > Any ideas? > > -Dave > > Btw, I'm using Camel 2.8.2 > > > > On Thu, Nov 12, 2015 at 12:43 PM, pmmerritt <pmmerr...@gmail.com> wrote: > >> We had a similar issue and traced it down to our usage of >> SftpChangedExclusiveReadLockStrategy which has a default check interval of >> 5 >> seconds which causes the polling to slow down waiting on files to finish >> being written, so max you can do is about 1 per 5 seconds. You can change >> the checkInterval so that it is faster, or we ended up writing our own read >> lock strategy that did not block the polling thread >> >> >> >> -- >> View this message in context: >> http://camel.465427.n5.nabble.com/sftp-endpoint-is-not-as-performant-as-expected-tp5773654p5773780.html >> Sent from the Camel - Users mailing list archive at Nabble.com. >> -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
