Johan Almqvist created CAMEL-25896:
--------------------------------------
Summary: camel-ftp - Let sendNoop=false also skip the consumer's
pre-poll noop (SFTP sends [email protected] on every poll)
Key: CAMEL-25896
URL: https://issues.apache.org/jira/browse/CAMEL-25896
Project: Camel
Issue Type: Improvement
Components: camel-ftp, camel-mina-sftp, camel-sftp
Affects Versions: 4.23.0, 4.18.5
Reporter: Johan Almqvist
h3. Problem
Before every poll, {{RemoteFileConsumer.connectIfNecessary()}} calls
{{getOperations().sendNoop()}}. For the JSch-based sftp component,
{{SftpOperations.sendNoop()}} calls {{Session.sendKeepAliveMsg()}}, which sends
a {{[email protected]}} global request. The {{sendNoop}} option (CAMEL-8807)
is read only by {{RemoteFileProducer}}, so a consumer cannot turn this off. The
code is the same in 4.18.5, in 4.23.0 and on main, where {{RemoteFileConsumer}}
now lives in camel-ftp-common.
ProFTPD's mod_sftp does not support that request. It replies with a failure and
writes one SFTPLog line for each one:
{noformat}
<timestamp> mod_sftp/<version>[<pid>]: client sent GLOBAL_REQUEST for
'[email protected]', denying
{noformat}
With the default {{delay=500}}, that is about two lines a second, roughly
170,000 a day, for each route polling the server. The volume grows with every
route, and it buries authentication failures and other real errors in the same
log. mod_sftp writes the line for every global request it does not handle (the
same code in 1.3.7 to 1.3.9, 1.3.10rc4 and master), so the server cannot
silence it without turning SFTPLog off entirely.
h3. For the JSch-based sftp component the keepalive does not change the outcome
* {{Session.sendKeepAliveMsg()}} writes the request and returns without waiting
for the reply, so it cannot tell a live server from a dead one.
* {{SftpOperations.sendNoop()}} only sends it when {{isConnected()}} is true.
Otherwise it returns false without sending anything.
* If the write throws, {{sendNoop()}} returns false and
{{connectIfNecessary()}} calls {{connect()}}. That returns at once while
{{isConnected()}} is still true, so nothing reconnects at that point. If JSch
has already marked the session disconnected, {{isConnected()}} alone triggers
the same reconnect.
* When the server closes the socket (the CAMEL-6591 case), JSch's reader thread
marks the session disconnected, so {{isConnected()}} already returns false and
{{sendNoop()}} sends nothing.
* A session that dies without the client noticing is recovered as it is today:
the poll fails, {{RemoteFilePollingConsumerPollStrategy.rollback()}}
force-disconnects, and the next poll reconnects.
So for an sftp consumer the result is the same as calling {{isConnected()}},
plus one packet per poll.
h3. Proposal
Let {{sendNoop}} also apply to the consumer's pre-poll check. With
{{sendNoop=false}}, the consumer relies on {{isConnected()}}:
{code:java}
protected void connectIfNecessary() {
boolean isConnected = false;
try {
if (getEndpoint().getConfiguration().isSendNoop()) {
isConnected = getOperations().sendNoop();
} else {
isConnected = getOperations().isConnected();
}
} catch (Exception ex) {
// unchanged
}
// unchanged from here
}
{code}
* The default stays {{true}}. The only users affected are those who already set
{{sendNoop=false}} on a consumer URI, where it is ignored today (for example a
URI shared between {{from()}} and {{to()}}). That is worth a line in the
upgrade guide.
* {{RemoteFileConsumer}} is shared, so this also applies to ftp, ftps and
mina-sftp. There {{sendNoop()}} is a real round trip (FTP {{NOOP}}, MINA
{{stat("/")}}). With {{sendNoop=false}}, a connection the server has closed is
found only when the poll fails, and the poll strategy then reconnects on the
next poll. That is the same trade-off {{sendNoop=false}} already makes for
producers.
* The {{@UriParam}} label changes from {{producer,advanced}} to
{{common,advanced}} (as {{fastExistsCheck}} uses). The description and the
{{setSendNoop()}} Javadoc cover the pre-poll check as well as the pre-write
check. Regenerating updates the component JSON, the catalog and the endpoint
DSL.
* Tests: a consumer counterpart of the existing producer {{sendNoop=false}}
test for ftp and sftp, and a unit test that with {{sendNoop=false}} the
consumer calls {{isConnected()}} instead of {{sendNoop()}} and still reconnects
when it returns false.
h3. Alternatives considered
* Going back to {{sendIgnore()}}: CAMEL-10149 moved away from it because
ProFTPD closed the connection on {{SSH_MSG_IGNORE}}.
* {{disconnect=true}}: no keepalive, but a full SSH handshake and login on
every poll.
* camel-mina-sftp: its {{sendNoop()}} is an SFTP {{stat("/")}} with no global
request, but switching components is not always an option.
* Changing {{SftpOperations.sendNoop()}} for everyone, either to return
{{isConnected()}} or to make a real round trip. That would change the default
for every SFTP user, so this issue proposes an opt-in that matches CAMEL-8807,
but either would also solve it.
h3. Workaround today
The only way to stop the keepalive today is to subclass {{SftpOperations}},
{{SftpEndpoint}} and {{SftpComponent}} and register the subclass as {{sftp}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
