Federico Mariani created CAMEL-26657:
----------------------------------------
Summary: camel-ftp, camel-sftp - stepwise cd per path segment and
back, and the autoCreate directory check, on every file
Key: CAMEL-26657
URL: https://issues.apache.org/jira/browse/CAMEL-26657
Project: Camel
Issue Type: Improvement
Components: camel-ftp, camel-sftp
Affects Versions: 4.18.5, 4.23.0, 4.22.2
Reporter: Federico Mariani
h3. Problem
With the default {{stepwise=true}}
({{components/camel-ftp-common/src/main/java/org/apache/camel/component/file/remote/RemoteFileConfiguration.java}}
line 89, main 7978db3d42dd), every upload and download saves the current
directory, changes into the target directory one path segment at a time, and
changes back afterwards:
* sftp (JSch): {{components/camel-ftp/.../SftpOperations.java}} {{storeFile}}
line 1271, {{retrieveFileToStreamInBody}} line 1100,
{{retrieveFileToFileInLocalWorkDirectory}} line 1156,
{{changeCurrentDirectory}} line 812. Each JSch {{cd}} is a REALPATH plus a STAT
(jsch 2.28.8 {{ChannelSftp.cd}}), although JSch keeps the working directory on
the client ({{pwd()}} sends nothing) and sends absolute paths anyway.
* ftp/ftps: {{FtpOperations.java}} {{storeFile}} line 691,
{{retrieveFileToStreamInBody}} line 468, {{changeCurrentDirectory}} line 905:
one CWD per segment and a PWD per operation. The way back is {{CWD /}} plus one
CWD per segment of the original directory.
On top of that, with the default {{autoCreate=true}} the producer runs
{{buildDirectory}} for every message
({{components/camel-file/.../GenericFileProducer.java}} lines 464-479):
* JSch with the default {{existDirCheckUsingLs=true}}: STAT, OPENDIR, one
READDIR batch, CLOSE ({{SftpOperations.java}} lines 697-756). With
{{existDirCheckUsingLs=false}}: a cd, then a stepwise cd back.
* FTP: PWD, CWD, CWD back ({{FtpOperations.java}} lines 387-416).
h3. How to reproduce
Embedded MINA SFTP server (JSch client) and embedded Apache FtpServer
(commons-net), counting SFTP requests with an {{SftpEventListener}} and FTP
commands with an Ftplet. Producer: 50 messages into {{/p1/out}} (2 segments),
counted after a warm-up message. Consumer: 20 files consumed ({{noop=true}},
{{maxMessagesPerPoll=20}}) from 1,000 files; consumer numbers include login and
the poll listing, divided by 20.
||Scenario||sftp (JSch) requests per message||ftp commands per message||
|producer defaults|15|10|
|{{stepwise=false}}|7|6|
|{{autoCreate=false}}|11|7|
|{{existDirCheckUsingLs=false}}|17|n/a|
|{{stepwise=false&autoCreate=false}} (ftp also {{sendNoop=false}})|3|2|
|consumer defaults, per file|15.1|6.85|
|consumer {{stepwise=false}}, per file|6.7|2.65|
One sftp upload with defaults (autoCreate check, cd p1, cd out, upload, cd ..,
cd ..):
{code}STAT OPENDIR READDIR CLOSE | REALPATH STAT REALPATH STAT | OPEN WRITE
CLOSE | REALPATH STAT REALPATH STAT{code}
One ftp upload with defaults:
{code}NOOP | PWD, CWD p1/out, CWD / | PWD, CWD p1, CWD out, PORT, STOR, CWD
/{code}
>From the code, each extra path segment adds 4 SFTP requests (cd in and back),
>and on ftp one more CWD in. The default {{sendNoop=true}} adds one NOOP per
>ftp message (two with a temp file name, as rename adds one); on sftp it is a
>keepalive packet with no round trip (CAMEL-25896).
h3. Proposed change
* sftp (JSch): track the working directory on the client and skip a cd into the
directory it is already in, or build the target path on the client as
{{stepwise=false}} does, since JSch sends absolute paths anyway. Keep the error
messages and the Windows drive path handling in {{changeCurrentDirectory}}
(CAMEL-12733, CAMEL-12848).
* ftp/ftps: keep stepwise (some servers need relative RETR/STOR, CAMEL-3309),
but track the working directory on the client (no PWD per file), and do not
change back between files of the same directory; restore once after the poll or
batch.
* {{autoCreate}}: cache the directories verified or created per connection;
clear the cache on disconnect; when a store fails, invalidate and retry once,
so a directory removed in the meantime is created again.
* Optional: skip the ftp NOOP when the connection was used a moment ago.
* Lower bound measured with {{stepwise=false&autoCreate=false}}: 3 SFTP
requests per upload (OPEN, WRITE, CLOSE) instead of 15, and 2 ftp commands
(PORT, STOR, with {{sendNoop=false}}) instead of 10.
h3. Impact / upgrade
* No option or default changes. The absolute paths the sftp server sees do not
change; the ftp command sequence does (fewer CWD/PWD), which is worth a note in
the 4.24 upgrade guide.
* camel-mina-sftp is not covered here (its stepwise handling differs and was
checked separately).
h3. Related
* CAMEL-3309 (introduced the stepwise option; stepwise is the deliberate
default, as some FTP servers require CD traversal), CAMEL-12733 (autoCreate
check moved from cd to ls to fix stepwise=false on Windows), CAMEL-14947
(existDirCheckUsingLs and the ls check that stops after the first batch),
CAMEL-12848 (stepwise cd on Windows), CAMEL-25896, CAMEL-26635.
* Code checked on camel-4.18.5, camel-4.22.2 and camel-4.23.0 (same stepwise
and autoCreate paths).
_Claude Code on behalf of Federico Mariani_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
