Federico Mariani created CAMEL-26635:
----------------------------------------
Summary: camel-mina-sftp, camel-sftp - stop listing the directory
once maxMessagesPerPoll files are accepted
Key: CAMEL-26635
URL: https://issues.apache.org/jira/browse/CAMEL-26635
Project: Camel
Issue Type: Improvement
Components: camel-sftp, camel-mina-sftp
Reporter: Federico Mariani
With {{maxMessagesPerPoll}} set (and {{eagerMaxMessagesPerPoll=true}}, the
default), the SFTP consumers still list the whole directory on every poll, and
only then stop picking files:
* {{AbstractSftpConsumer.doPollDirectory}} gets the full array from
{{operations.listFiles(dir)}}; {{canPollMoreFiles}} only stops the loop over it.
* {{MinaSftpOperations.listFiles}} drains {{sftpClient.readDir(path)}};
{{SftpOperations.listFiles}} calls {{channel.ls(path)}}.
The option description says "If eager then the limit is during the scanning of
files", so users expect the listing itself to stop at the limit.
h3. Impact
Measured with the embedded MINA SFTP server, 30,000 files,
{{maxMessagesPerPoll=1000}}, {{stepwise=false}}:
||Listing||Entries read in one poll||READDIR requests||
|today (whole directory)|30,002|145|
|stop once 1,000 files are accepted|about 1,040|5|
30,002 is the files plus {{.}} and {{..}}; about 1,040 because the server sends
entries in batches, so the last batch goes past 1,000.
With {{delete=true}} and {{greedy=true}}, every poll re-reads what is left:
draining 25,000 files reads about 325,000 entries (25k + 24k + ... + 1k)
instead of about 25,000.
A user hit this with camel-mina-sftp and camel-sftp on Azure Blob Storage SFTP
(a folder of 25k+ files, {{maxMessagesPerPoll=1000}}, {{stepwise=false}}): the
listing gets cut, and Azure closes connections that are idle for two minutes.
It is not known yet whether a shorter listing avoids the cut there (it depends
on where Azure spends the time; keep-alive with {{serverAliveInterval}} is
being tried first), but reading only what a poll needs is worth it regardless.
h3. Proposal
Stop reading the directory once {{maxMessagesPerPoll}} files are *accepted* (as
counted by {{canPollMoreFiles}}, after include/exclude, filters, idempotent and
in-progress checks), rather than capping the raw listing at
{{maxMessagesPerPoll}} entries.
A raw cap (for example {{listFiles(path, maxMessages)}}) starves when rejected
entries stay in the directory, because they keep filling the cap: files not
matching {{include}}, files already in the idempotent repository,
{{noop=true}}. Reproduced: with {{noop=true}}, 300 files and
{{maxMessagesPerPoll=100}}, a raw cap consumes 98 files and then nothing more
({{.}} and {{..}} count against the cap too); stopping on accepted files
consumes all 300.
No new option: use it whenever the outcome is the same as today, and keep the
full listing otherwise:
* {{maxMessagesPerPoll > 0}} and {{eagerMaxMessagesPerPoll=true}}
* no {{preSort}} (it sorts the whole listing, CAMEL-23931)
* no {{doneFileName}} ({{isMatched}} looks the done file up in the listing; see
also CAMEL-25728 and CAMEL-25772)
* both {{stepwise=true}} (the default) and {{stepwise=false}}
Nothing is kept between polls: each poll lists from the start, as today. With
{{delete}} or {{move}} a poll reads about {{maxMessagesPerPoll}} entries; with
{{noop}} or idempotent it reads further on each poll, up to the whole directory.
h3. Implementation notes
* camel-mina-sftp: use {{sftpClient.openDir(path)}} and {{listDir(handle)}} in
try-with-resources. {{readDir(path)}} closes the handle only at the end of the
directory, so leaving a loop over it early leaves the handle open.
* camel-sftp (JSch): {{ChannelSftp.ls(path, LsEntrySelector)}} can stop with
{{BREAK}}, but an SFTP request sent from inside the selector corrupts the
channel (reproduced: a {{stat}} in the selector makes {{ls}} fail with
{{ArrayIndexOutOfBoundsException}}). The consumer sends requests while it walks
the listing (resolving symbolic links, recursion), so camel-sftp needs either a
two-step approach or only the cases that send none. Suggest doing
camel-mina-sftp first.
* Test: count the READDIR requests on the embedded server (with an
{{SftpEventListener}}) instead of measuring time.
h3. Same pattern elsewhere (not in scope)
* camel-smb: {{SmbOperations.listFiles}} returns {{share.list(path,
searchPattern)}}, the whole directory; smbj 0.15.0 also offers a lazy
{{Directory.iterator(...)}}.
* camel-file: {{File.listFiles()}} reads the whole directory, while
{{Files.newDirectoryStream}} is lazy. Mostly matters on network file systems.
* camel-ftp: {{LIST}} is a single data transfer, so there is little to gain.
_Claude Code on behalf of Federico Mariani_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
