I'm rather new to Camel, I'm assuming that if the SFTP source gets 900 files then the configured GenericFileExclusiveReadLockStrategy#acquireExclusiveReadLock will get called 900 times (once per file) every polling cycle.
We too have a custom GenericFileExclusiveReadLockStrategy class that does not block. What ours does is maintain a map/cache of of the lastModifiedTime and fileSize for each of the 900 files. The call to acquireExclusiveReadLock() will return true as soon as neither of those values changed since the last poll cycle. We have our delay/poll cycle set to 60 seconds. I'm not sure why such a large value but we are fine waiting 2 minutes to get each of the 900 files...and by the time 2 minutes go by we will likely have more files in the SFTP source. E.g. if we have a constant delay in receiving each file of 2-3 minutes that is more than fine. The problem is that on each cycle it only processes 3-5 files instead of the 900 that should have returned true from acquireExclusiveReadLock() What seems to be happening is that Camel is not calling the acquireExclusiveReadLock() method for each of the 900 files every 60 seconds, rather it slows down and either calls just a few of those 900 files or none. Any ideas? -Dave Btw, I'm using Camel 2.8.2 On Thu, Nov 12, 2015 at 12:43 PM, pmmerritt <pmmerr...@gmail.com> wrote: > We had a similar issue and traced it down to our usage of > SftpChangedExclusiveReadLockStrategy which has a default check interval of > 5 > seconds which causes the polling to slow down waiting on files to finish > being written, so max you can do is about 1 per 5 seconds. You can change > the checkInterval so that it is faster, or we ended up writing our own read > lock strategy that did not block the polling thread > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/sftp-endpoint-is-not-as-performant-as-expected-tp5773654p5773780.html > Sent from the Camel - Users mailing list archive at Nabble.com. >
