This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new ff82c3dd1bde CAMEL-26999, CAMEL-27000: camel-ftp - a server that stops
answering no longer holds the stop, the failed poll and the rest of the batch
for soTimeout (#30087)
ff82c3dd1bde is described below
commit ff82c3dd1bde078905ebbcd5e0a76c4372dda633
Author: Federico Mariani <[email protected]>
AuthorDate: Sun Nov 15 23:03:42 2026 +0100
CAMEL-26999, CAMEL-27000: camel-ftp - a server that stops answering no
longer holds the stop, the failed poll and the rest of the batch for soTimeout
(#30087)
* CAMEL-26999: camel-ftp - a disconnect no longer waits up to soTimeout…
… for the reply to QUIT
FtpOperations.doDisconnect() called FTPClient.logout(), which waits for the
reply to QUIT up to soTimeout (5 minutes by default), and forceDisconnect()
called the same method. When the server stopped answering on an open
connection, the stop of an ftp consumer or producer held CamelContext.stop()
for soTimeout, and a failed poll waited soTimeout once more before the next
poll could connect.
The forced disconnect now closes the connection without QUIT, and a
disconnect
waits for the reply to QUIT at most 2 seconds (soTimeout when shorter).
* CAMEL-27000: camel-ftp - after a read timeout the consumer leaves the…
… rest of the batch to the next poll
When the server stopped answering while the ftp consumer retrieved a file,
the file failed after soTimeout (or timeout during the transfer), and the
consumer went on with the next file of the batch on the same connection:
each remaining file sent its first command on the dead connection and waited
soTimeout, and the next poll waited once more on its NOOP. commons-net also
replaced the socket of the dead connection without closing it.
After a file failed with a SocketTimeoutException, the consumer now closes
the
connection without waiting on it and ends the batch; the next poll connects
again and picks up the remaining files. FtpOperations closes a connection
that
is still open before connecting again.
* CAMEL-27000: camel-ftp - with streamDownload, a timeout at the end of…
… the transfer also leaves the rest of the batch to the next poll
With streamDownload=true the route reads the file, and the reply that ends
the transfer is read by
releaseRetrievedFileResources() in the commit or rollback of the file,
whose failure goes to the
on-completion exception handler, not to the consumer. When that reply timed
out, the next file of
the batch sent PASV on the connection that stopped answering, waited
soTimeout and failed.
FtpOperations.releaseRetrievedFileResources() now closes the connection
without QUIT when that
reply times out, and FtpConsumer.isBatchAllowed() ends the batch when the
connection is closed.
* CAMEL-26999: camel-ftp - upgrade guide: a server slower than 2 s to a…
…nswer QUIT makes the disconnect fail
A producer with disconnect=true or disconnectOnBatchComplete=true logs it
as a WARN where it used
to wait for the reply; the consumer and the stop log it at DEBUG.
Co-Authored-By: Claude <[email protected]>
---
.../apache/camel/catalog/docs/ftp-component.adoc | 15 +
.../camel-ftp/src/main/docs/ftp-component.adoc | 15 +
.../camel/component/file/remote/FtpConsumer.java | 34 ++
.../camel/component/file/remote/FtpOperations.java | 57 ++-
.../camel/component/file/remote/FakeFtpServer.java | 395 +++++++++++++++++++++
.../FtpConsumerBatchAfterReadTimeoutTest.java | 182 ++++++++++
.../remote/FtpDisconnectFromSilentServerTest.java | 137 +++++++
.../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc | 21 ++
8 files changed, 853 insertions(+), 3 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/ftp-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/ftp-component.adoc
index 5e51f1f747a1..23a2866e4912 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/ftp-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/ftp-component.adoc
@@ -253,6 +253,21 @@ It is only a single endpoint that does not support
concurrent consumers.
The FTP producer does *not* have this issue, it supports concurrency.
+=== A server that stops answering
+
+When the FTP/FTPS server stops answering on an open connection (it hangs, or a
firewall drops the connection without
+closing it), the client notices it only when `soTimeout` expires (or `timeout`
during a transfer). Camel then does not
+wait on that connection again:
+
+* When the consumer retrieves a file and the server does not answer in time,
only that file fails: the consumer closes
+ the connection, which it cannot use any more, and leaves the other files of
the poll to the next poll, which connects
+ again. With `streamDownload=true`, the route reads the file from the server:
when that read times out, the file fails
+ in the route, and the consumer waits up to `soTimeout` for the reply that
ends the transfer, then closes the
+ connection in the same way.
+* After a failed poll, the consumer closes the connection without sending
`QUIT`.
+* When disconnecting, for example when the consumer or the producer stops, the
client sends `QUIT` and waits at most
+ 2 seconds (`soTimeout` when it is shorter) for its reply.
+
[[FTP-DefaultWhenConsumingFiles]]
=== Default when consuming files
diff --git a/components/camel-ftp/src/main/docs/ftp-component.adoc
b/components/camel-ftp/src/main/docs/ftp-component.adoc
index 5e51f1f747a1..23a2866e4912 100644
--- a/components/camel-ftp/src/main/docs/ftp-component.adoc
+++ b/components/camel-ftp/src/main/docs/ftp-component.adoc
@@ -253,6 +253,21 @@ It is only a single endpoint that does not support
concurrent consumers.
The FTP producer does *not* have this issue, it supports concurrency.
+=== A server that stops answering
+
+When the FTP/FTPS server stops answering on an open connection (it hangs, or a
firewall drops the connection without
+closing it), the client notices it only when `soTimeout` expires (or `timeout`
during a transfer). Camel then does not
+wait on that connection again:
+
+* When the consumer retrieves a file and the server does not answer in time,
only that file fails: the consumer closes
+ the connection, which it cannot use any more, and leaves the other files of
the poll to the next poll, which connects
+ again. With `streamDownload=true`, the route reads the file from the server:
when that read times out, the file fails
+ in the route, and the consumer waits up to `soTimeout` for the reply that
ends the transfer, then closes the
+ connection in the same way.
+* After a failed poll, the consumer closes the connection without sending
`QUIT`.
+* When disconnecting, for example when the consumer or the producer stops, the
client sends `QUIT` and waits at most
+ 2 seconds (`soTimeout` when it is shorter) for its reply.
+
[[FTP-DefaultWhenConsumingFiles]]
=== Default when consuming files
diff --git
a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpConsumer.java
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpConsumer.java
index 79f8ea5b087c..56ebf5d92ddb 100644
---
a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpConsumer.java
+++
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpConsumer.java
@@ -16,6 +16,7 @@
*/
package org.apache.camel.component.file.remote;
+import java.net.SocketTimeoutException;
import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
@@ -290,6 +291,39 @@ public class FtpConsumer extends
RemoteFileConsumer<FTPFile> {
return false;
}
+ /**
+ * A file failed because the server did not answer in time (soTimeout on
the control connection, or timeout on the
+ * data connection): the server may no longer answer, and a reply that
comes late would be read as the reply to the
+ * next command, so the connection cannot be used for the rest of the
batch. It is closed without waiting on it, and
+ * the rest of the batch is left to the next poll, which connects again.
Otherwise each remaining file would wait
+ * soTimeout on the connection and fail.
+ */
+ @Override
+ protected void handleException(String message, Exchange exchange,
Throwable t) {
+ super.handleException(message, exchange, t);
+ if (loggedIn &&
ObjectHelper.getException(SocketTimeoutException.class, t) != null) {
+ LOG.debug("Closing the connection to {} after a timeout, the rest
of the batch is left to the next poll",
+ remoteServer());
+ forceDisconnect();
+ }
+ }
+
+ /**
+ * The batch goes on only while the connection is open. It is closed after
the server did not answer in time: by
+ * {@link #handleException(String, Exchange, Throwable)} when the consumer
retrieves the file, or by
+ * {@link FtpOperations#releaseRetrievedFileResources(Exchange)} at the
end of a file streamed to the route
+ * (streamDownload=true). The rest of the batch is then left to the next
poll, which connects again.
+ */
+ @Override
+ public boolean isBatchAllowed() {
+ if (!loggedIn || !getOperations().isConnected()) {
+ // the rest of the batch is not processed
+ pendingExchanges = 0;
+ return false;
+ }
+ return super.isBatchAllowed();
+ }
+
@Override
protected boolean ignoreCannotRetrieveFile(String name, Exchange exchange,
Exception cause) {
// the client may still have the reply code 550 of an earlier command
when the failure is a connection that the
diff --git
a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java
index 572f426f3a75..fe5f3d9b0e3e 100644
---
a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java
+++
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java
@@ -23,8 +23,10 @@ import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
+import java.net.SocketTimeoutException;
import java.time.Duration;
import java.util.Iterator;
+import java.util.Objects;
import java.util.concurrent.ScheduledExecutorService;
import org.apache.camel.Exchange;
@@ -56,6 +58,11 @@ import org.slf4j.LoggerFactory;
*/
public class FtpOperations implements RemoteFileOperations<FTPFile> {
+ /**
+ * How long a disconnect waits at most for the reply to QUIT (soTimeout
when it is shorter).
+ */
+ private static final int QUIT_REPLY_TIMEOUT = 2000;
+
protected final Logger log = LoggerFactory.getLogger(getClass());
protected final FTPClient client;
protected final FTPClientConfig clientConfig;
@@ -118,6 +125,17 @@ public class FtpOperations implements
RemoteFileOperations<FTPFile> {
String username = configuration.getUsername();
String account = ((FtpConfiguration) configuration).getAccount();
+ // connecting replaces the socket of the client without closing it:
close a connection that is still open, such
+ // as one whose server no longer answers, so it is not left open
+ if (client.isConnected()) {
+ log.trace("Client disconnect before connecting again");
+ try {
+ client.disconnect();
+ } catch (IOException e) {
+ log.trace("Ignore exception during disconnect: {}",
e.getMessage());
+ }
+ }
+
if (clientConfig != null) {
log.trace("Configuring FTPClient with config: {}", clientConfig);
client.configure(clientConfig);
@@ -157,9 +175,10 @@ public class FtpOperations implements
RemoteFileOperations<FTPFile> {
if (payload.exception instanceof
GenericFileOperationFailedException genericFileOperationFailedException) {
throw genericFileOperationFailedException;
} else {
+ // no reply string when the client has no reply, such
as after it closed the previous connection
throw new GenericFileOperationFailedException(
- client.getReplyCode(),
client.getReplyString(), payload.exception.getMessage(),
- payload.exception);
+ client.getReplyCode(),
Objects.toString(client.getReplyString(), ""),
+ payload.exception.getMessage(),
payload.exception);
}
} else {
throw new GenericFileOperationFailedException(
@@ -303,9 +322,20 @@ public class FtpOperations implements
RemoteFileOperations<FTPFile> {
}
}
+ /**
+ * Closes the connection without sending QUIT: a forced disconnect
recovers from a failure, after which the server
+ * may no longer answer, so it must not wait up to soTimeout for the reply.
+ */
@Override
public void forceDisconnect() throws GenericFileOperationFailedException {
- doDisconnect();
+
clientActivityListener.onDisconnecting(endpoint.getConfiguration().remoteServerInformation());
+ try {
+ log.trace("Client disconnect");
+ client.disconnect();
+ } catch (IOException e) {
+ throw new
GenericFileOperationFailedException(client.getReplyCode(),
client.getReplyString(), e.getMessage(), e);
+ }
+
clientActivityListener.onDisconnected(endpoint.getConfiguration().remoteServerInformation());
}
protected void doDisconnect() throws GenericFileOperationFailedException {
@@ -313,6 +343,15 @@ public class FtpOperations implements
RemoteFileOperations<FTPFile> {
clientActivityListener.onDisconnecting(endpoint.getConfiguration().remoteServerInformation());
try {
log.trace("Client logout");
+ // the connection is closed anyway, so a server that no longer
answers (it hangs, or the connection was lost
+ // without being closed) must not hold the disconnect, such as the
stop of the consumer or producer, for
+ // soTimeout while waiting for the reply to QUIT
+ if (client.isConnected()) {
+ int soTimeout = client.getSoTimeout();
+ if (soTimeout <= 0 || soTimeout > QUIT_REPLY_TIMEOUT) {
+ client.setSoTimeout(QUIT_REPLY_TIMEOUT);
+ }
+ }
client.logout();
client.disconnect();
} catch (IOException e) {
@@ -458,6 +497,18 @@ public class FtpOperations implements
RemoteFileOperations<FTPFile> {
throw new GenericFileOperationFailedException(
client.getReplyCode(), client.getReplyString());
}
+ } catch (SocketTimeoutException e) {
+ // the server did not answer in time at the end of the
streamed transfer: a reply that comes late would be
+ // read as the reply to the next command, so the connection
cannot be used any more. It is closed
+ // without waiting on it, and the consumer leaves the rest of
the batch to the next poll, which connects
+ // again
+ GenericFileOperationFailedException failure = new
GenericFileOperationFailedException(e.getMessage(), e);
+ try {
+ forceDisconnect();
+ } catch (GenericFileOperationFailedException fe) {
+ failure.addSuppressed(fe);
+ }
+ throw failure;
} catch (IOException e) {
throw new GenericFileOperationFailedException(e.getMessage(),
e);
}
diff --git
a/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FakeFtpServer.java
b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FakeFtpServer.java
new file mode 100644
index 000000000000..c84710caac1a
--- /dev/null
+++
b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FakeFtpServer.java
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.file.remote;
+
+import java.io.BufferedReader;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.net.ServerSocket;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
+
+/**
+ * A minimal in-memory FTP server (passive mode only) that can stop answering
on a connection that stays open, as a
+ * server that hangs, or a network path that is lost without closing the
connection, does.
+ * <p>
+ * It serves the commands the ftp client sends with the default options and
{@code passiveMode=true}: login, SYST, TYPE,
+ * PWD, CWD, CDUP, MKD, NOOP, PASV, LIST, NLST, RETR, STOR, DELE, RNFR/RNTO,
SIZE, MDTM and QUIT. A silent connection
+ * keeps reading and records every command it receives, but never answers
again. New connections are served normally, so
+ * a client that opens a new connection recovers.
+ */
+final class FakeFtpServer implements Closeable {
+
+ /** How the server stops answering when the trigger command arrives. */
+ enum Stall {
+ /** The trigger command gets no reply: the client waits for the reply
on the control connection. */
+ BEFORE_REPLY,
+ /**
+ * The trigger RETR gets its 150 reply and half of the file on the
data connection, then nothing more on either
+ * connection: the client waits on the data connection.
+ */
+ MID_TRANSFER
+ }
+
+ private final ServerSocket serverSocket;
+ private final ExecutorService executor = Executors.newCachedThreadPool(r
-> {
+ Thread t = new Thread(r, "fake-ftp-server");
+ t.setDaemon(true);
+ return t;
+ });
+ private final Map<String, byte[]> files = new ConcurrentHashMap<>();
+ private final Set<String> directories = ConcurrentHashMap.newKeySet();
+ private final List<Connection> connections = new CopyOnWriteArrayList<>();
+ private final AtomicInteger connectionIds = new AtomicInteger();
+ private volatile Predicate<String> trigger;
+ private volatile Stall stall = Stall.BEFORE_REPLY;
+ private volatile boolean closed;
+
+ FakeFtpServer() throws IOException {
+ serverSocket = new ServerSocket(0, 50,
InetAddress.getLoopbackAddress());
+ directories.add("/");
+ executor.execute(this::acceptLoop);
+ }
+
+ int port() {
+ return serverSocket.getLocalPort();
+ }
+
+ void addFile(String path, byte[] content) {
+ String file = normalize(path);
+ String parent = parentOf(file);
+ while (!parent.equals("/")) {
+ directories.add(parent);
+ parent = parentOf(parent);
+ }
+ files.put(file, content);
+ }
+
+ Map<String, byte[]> files() {
+ return files;
+ }
+
+ /**
+ * The first connection that receives a command matching {@code command}
stops answering, from that command on.
+ */
+ void stallOn(Predicate<String> command, Stall how) {
+ this.stall = how;
+ this.trigger = command;
+ }
+
+ /** Every open connection stops answering from now on. New connections are
served. */
+ void silenceOpenConnections() {
+ connections.forEach(c -> c.silent = true);
+ }
+
+ /** The commands the client sent on a connection after that connection
stopped answering. */
+ List<String> commandsOnSilentConnections() {
+ List<String> answer = new ArrayList<>();
+ connections.forEach(c -> answer.addAll(c.unanswered));
+ return answer;
+ }
+
+ /** How many control connections the client has opened so far. */
+ int connectionsOpened() {
+ return connectionIds.get();
+ }
+
+ /** How many connections that stopped answering the client has not closed.
*/
+ long silentConnectionsOpen() {
+ return connections.stream().filter(c -> c.silent && !c.ended).count();
+ }
+
+ @Override
+ public void close() throws IOException {
+ closed = true;
+ serverSocket.close();
+ for (Connection c : connections) {
+ c.close();
+ }
+ executor.shutdownNow();
+ }
+
+ private void acceptLoop() {
+ while (!closed) {
+ try {
+ Socket socket = serverSocket.accept();
+ Connection c = new Connection(socket,
connectionIds.incrementAndGet());
+ connections.add(c);
+ executor.execute(c);
+ } catch (IOException e) {
+ // closed
+ return;
+ }
+ }
+ }
+
+ private final class Connection implements Runnable {
+
+ private final Socket socket;
+ private final int id;
+ private final List<String> unanswered = new CopyOnWriteArrayList<>();
+ private volatile boolean silent;
+ private volatile boolean ended;
+ private ServerSocket passive;
+ // a data connection left open by a stalled transfer
+ private Socket stalledData;
+ private OutputStream out;
+ private String cwd = "/";
+ private String renameFrom;
+
+ Connection(Socket socket, int id) {
+ this.socket = socket;
+ this.id = id;
+ }
+
+ @Override
+ public void run() {
+ try {
+ BufferedReader in = new BufferedReader(
+ new InputStreamReader(socket.getInputStream(),
StandardCharsets.UTF_8));
+ out = socket.getOutputStream();
+ reply("220 fake FTP server ready (connection " + id + ")");
+ String line;
+ while ((line = in.readLine()) != null) {
+ Predicate<String> t = trigger;
+ if (!silent && t != null && t.test(line)) {
+ // one shot: only the first matching connection stalls
+ trigger = null;
+ if (stall == Stall.MID_TRANSFER &&
line.startsWith("RETR ")) {
+ startTransferThenStall(line.substring(5).trim());
+ }
+ silent = true;
+ continue;
+ }
+ if (silent) {
+ unanswered.add(line);
+ continue;
+ }
+ if (!handle(line)) {
+ break;
+ }
+ }
+ } catch (IOException e) {
+ // the client or the test closed the connection
+ } finally {
+ ended = true;
+ close();
+ }
+ }
+
+ private boolean handle(String line) throws IOException {
+ int space = line.indexOf(' ');
+ String cmd = (space < 0 ? line : line.substring(0,
space)).toUpperCase();
+ String arg = space < 0 ? "" : line.substring(space + 1).trim();
+ switch (cmd) {
+ case "USER" -> reply("331 Password required");
+ case "PASS" -> reply("230 Logged in");
+ case "SYST" -> reply("215 UNIX Type: L8");
+ case "TYPE", "MODE", "STRU", "OPTS", "SITE", "NOOP" ->
reply("200 OK");
+ case "PWD", "XPWD" -> reply("257 \"" + cwd + "\" is the
current directory");
+ case "CWD" -> {
+ String dir = resolve(arg);
+ if (directories.contains(dir)) {
+ cwd = dir;
+ reply("250 OK");
+ } else {
+ reply("550 No such directory");
+ }
+ }
+ case "CDUP" -> {
+ cwd = parentOf(cwd);
+ reply("250 OK");
+ }
+ case "MKD" -> {
+ directories.add(resolve(arg));
+ reply("257 \"" + resolve(arg) + "\" created");
+ }
+ case "PASV" -> {
+ closePassive();
+ passive = new ServerSocket(0, 1,
InetAddress.getLoopbackAddress());
+ passive.setSoTimeout(10000);
+ int p = passive.getLocalPort();
+ reply("227 Entering Passive Mode (127,0,0,1," + (p >> 8) +
"," + (p & 0xff) + ")");
+ }
+ case "LIST", "NLST" -> list(cmd, arg);
+ case "RETR" -> retrieve(arg);
+ case "STOR" -> store(arg);
+ case "DELE" -> reply(files.remove(resolve(arg)) != null ? "250
Deleted" : "550 No such file");
+ case "RNFR" -> {
+ renameFrom = resolve(arg);
+ reply(files.containsKey(renameFrom) ? "350 Ready for RNTO"
: "550 No such file");
+ }
+ case "RNTO" -> {
+ byte[] data = renameFrom != null ?
files.remove(renameFrom) : null;
+ if (data != null) {
+ addFile(resolve(arg), data);
+ reply("250 Renamed");
+ } else {
+ reply("550 No such file");
+ }
+ }
+ case "SIZE" -> {
+ byte[] data = files.get(resolve(arg));
+ reply(data != null ? "213 " + data.length : "550 No such
file");
+ }
+ case "MDTM" -> reply(files.containsKey(resolve(arg)) ? "213
20250101000000" : "550 No such file");
+ case "QUIT" -> {
+ reply("221 Bye");
+ return false;
+ }
+ default -> reply("502 Command not implemented");
+ }
+ return true;
+ }
+
+ private void list(String cmd, String arg) throws IOException {
+ String dir = cwd;
+ for (String token : arg.split(" ")) {
+ if (!token.isEmpty() && !token.startsWith("-")) {
+ dir = resolve(token);
+ }
+ }
+ StringBuilder listing = new StringBuilder();
+ for (Map.Entry<String, byte[]> e : new
TreeMap<>(files).entrySet()) {
+ if (parentOf(e.getKey()).equals(dir)) {
+ String name =
e.getKey().substring(e.getKey().lastIndexOf('/') + 1);
+ if (cmd.equals("NLST")) {
+ listing.append(name).append("\r\n");
+ } else {
+ listing.append("-rw-r--r-- 1 camel camel
").append(e.getValue().length)
+ .append(" Jan 01 2025
").append(name).append("\r\n");
+ }
+ }
+ }
+ transfer(listing.toString().getBytes(StandardCharsets.UTF_8));
+ }
+
+ private void retrieve(String arg) throws IOException {
+ byte[] data = files.get(resolve(arg));
+ if (data == null) {
+ reply("550 No such file");
+ return;
+ }
+ transfer(data);
+ }
+
+ private void transfer(byte[] data) throws IOException {
+ reply("150 Opening data connection");
+ try (Socket d = passive.accept(); OutputStream o =
d.getOutputStream()) {
+ o.write(data);
+ } finally {
+ closePassive();
+ }
+ reply("226 Transfer complete");
+ }
+
+ private void startTransferThenStall(String arg) throws IOException {
+ byte[] data = files.get(resolve(arg));
+ reply("150 Opening data connection");
+ stalledData = passive.accept();
+ OutputStream o = stalledData.getOutputStream();
+ o.write(data, 0, data.length / 2);
+ o.flush();
+ // the rest of the file and the 226 reply never come, and the data
connection stays open
+ }
+
+ private void store(String arg) throws IOException {
+ reply("150 Opening data connection");
+ ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ try (Socket d = passive.accept(); InputStream i =
d.getInputStream()) {
+ i.transferTo(bos);
+ } finally {
+ closePassive();
+ }
+ addFile(resolve(arg), bos.toByteArray());
+ reply("226 Transfer complete");
+ }
+
+ private String resolve(String path) {
+ if (path.isEmpty()) {
+ return cwd;
+ }
+ return normalize(path.startsWith("/") ? path : cwd + "/" + path);
+ }
+
+ private void reply(String line) throws IOException {
+ out.write((line + "\r\n").getBytes(StandardCharsets.UTF_8));
+ out.flush();
+ }
+
+ private void closePassive() {
+ if (passive != null) {
+ try {
+ passive.close();
+ } catch (IOException e) {
+ // ignore
+ }
+ passive = null;
+ }
+ }
+
+ void close() {
+ closePassive();
+ try {
+ if (stalledData != null) {
+ stalledData.close();
+ }
+ socket.close();
+ } catch (IOException e) {
+ // ignore
+ }
+ }
+ }
+
+ static String normalize(String path) {
+ List<String> parts = new ArrayList<>();
+ for (String p : path.split("/")) {
+ if (p.isEmpty() || p.equals(".")) {
+ continue;
+ }
+ if (p.equals("..")) {
+ if (!parts.isEmpty()) {
+ parts.remove(parts.size() - 1);
+ }
+ } else {
+ parts.add(p);
+ }
+ }
+ return "/" + String.join("/", parts);
+ }
+
+ static String parentOf(String path) {
+ int i = path.lastIndexOf('/');
+ return i <= 0 ? "/" : path.substring(0, i);
+ }
+}
diff --git
a/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FtpConsumerBatchAfterReadTimeoutTest.java
b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FtpConsumerBatchAfterReadTimeoutTest.java
new file mode 100644
index 000000000000..3b1f975ccbfe
--- /dev/null
+++
b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FtpConsumerBatchAfterReadTimeoutTest.java
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.file.remote;
+
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.camel.Exchange;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.spi.ExceptionHandler;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.EnumSource;
+
+import static org.awaitility.Awaitility.await;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * When the FTP server stops answering on the connection of the consumer in
the middle of a batch, only the file being
+ * transferred fails: the consumer closes the connection, which cannot be used
any more, and leaves the rest of the
+ * batch to the next poll, which connects again. Before, every remaining file
of the batch sent a command on the dead
+ * connection and waited soTimeout for its reply. The same holds when the file
is streamed to the route
+ * (streamDownload=true).
+ */
+public class FtpConsumerBatchAfterReadTimeoutTest extends CamelTestSupport {
+
+ private static final int FILES = 10;
+
+ private final RecordingExceptionHandler failures = new
RecordingExceptionHandler();
+ private FakeFtpServer server;
+
+ @BeforeEach
+ void startServer() throws Exception {
+ server = new FakeFtpServer();
+ context.getRegistry().bind("failures", failures);
+ }
+
+ @AfterEach
+ void stopServer() throws Exception {
+ server.close();
+ }
+
+ @ParameterizedTest
+ @EnumSource(FakeFtpServer.Stall.class)
+ void readTimeoutFailsOnlyTheFileBeingTransferred(FakeFtpServer.Stall
stall) throws Exception {
+ for (int i = 1; i <= FILES; i++) {
+ server.addFile(String.format("/inbox/f-%02d.txt", i), ("content of
file " + i).getBytes(StandardCharsets.UTF_8));
+ }
+ // the connection stops answering at the 3rd file: before the reply to
RETR (the client times out on the
+ // control connection), or in the middle of the transfer (the client
times out on the data connection)
+ server.stallOn(command -> command.equals("RETR f-03.txt"), stall);
+
+ Map<String, String> delivered = new ConcurrentHashMap<>();
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ from("ftp://[email protected]:" + server.port() +
"/inbox?password=secret&passiveMode=true"
+ +
"&noop=true&sortBy=file:name&initialDelay=0&delay=100&soTimeout=1000&timeout=1000"
+ + "&exceptionHandler=#failures")
+ .process(e ->
delivered.put(e.getMessage().getHeader(Exchange.FILE_NAME, String.class),
+ e.getMessage().getBody(String.class)));
+ }
+ });
+
+ await().atMost(30, TimeUnit.SECONDS).until(() -> delivered.size() ==
FILES);
+
+ for (int i = 1; i <= FILES; i++) {
+ assertEquals("content of file " + i,
delivered.get(String.format("f-%02d.txt", i)));
+ }
+ assertEquals(List.of("f-03.txt"), failures.files, "the files that
failed");
+ // no file of the batch waited soTimeout for a reply on the connection
that stopped answering
+ assertEquals(List.of(), server.commandsOnSilentConnections(), "the
commands sent after the stall");
+ await().atMost(5, TimeUnit.SECONDS).until(() ->
server.silentConnectionsOpen() == 0);
+ }
+
+ @ParameterizedTest
+ @EnumSource(FakeFtpServer.Stall.class)
+ void streamedReadTimeoutFailsOnlyTheFileBeingRead(FakeFtpServer.Stall
stall) throws Exception {
+ for (int i = 1; i <= FILES; i++) {
+ server.addFile(String.format("/inbox/f-%02d.txt", i), ("content of
file " + i).getBytes(StandardCharsets.UTF_8));
+ }
+ // with streamDownload the route reads the file from the data
connection: the connection stops answering at the
+ // 3rd file, before the reply to RETR (the consumer fails to retrieve
the file), or in the middle of the transfer
+ // (the route fails to read the file, then the consumer waits for the
reply that ends the transfer)
+ server.stallOn(command -> command.equals("RETR inbox/f-03.txt"),
stall);
+
+ Map<String, String> delivered = new ConcurrentHashMap<>();
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ onException(Exception.class)
+ .process(e ->
failures.files.add(e.getMessage().getHeader(Exchange.FILE_NAME_ONLY,
String.class)));
+
+ from("ftp://[email protected]:" + server.port() +
"/inbox?password=secret&passiveMode=true"
+ +
"&noop=true&sortBy=file:name&initialDelay=0&delay=100&soTimeout=1000&timeout=1000"
+ +
"&streamDownload=true&stepwise=false&exceptionHandler=#failures")
+ .process(e ->
delivered.put(e.getMessage().getHeader(Exchange.FILE_NAME, String.class),
+ e.getMessage().getBody(String.class)));
+ }
+ });
+
+ await().atMost(30, TimeUnit.SECONDS).until(() -> delivered.size() ==
FILES);
+
+ for (int i = 1; i <= FILES; i++) {
+ assertEquals("content of file " + i,
delivered.get(String.format("f-%02d.txt", i)));
+ }
+ // in the route or in the consumer
+ assertEquals(List.of("f-03.txt"), failures.files, "the files that
failed");
+ // the next file of the batch did not wait soTimeout for a reply on
the connection that stopped answering
+ assertEquals(List.of(), server.commandsOnSilentConnections(), "the
commands sent after the stall");
+ await().atMost(5, TimeUnit.SECONDS).until(() ->
server.silentConnectionsOpen() == 0);
+ }
+
+ @Test
+ void connectionThatStoppedAnsweringIsClosedBeforeConnectingAgain() throws
Exception {
+ server.addFile("/inbox/a.txt",
"hello".getBytes(StandardCharsets.UTF_8));
+ List<String> received = new CopyOnWriteArrayList<>();
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ from("ftp://[email protected]:" + server.port() +
"/inbox?password=secret&passiveMode=true"
+ + "&noop=true&initialDelay=0&delay=1000&soTimeout=1000")
+ .process(e ->
received.add(e.getMessage().getHeader(Exchange.FILE_NAME, String.class)));
+ }
+ });
+ await().atMost(10, TimeUnit.SECONDS).until(() -> received.size() == 1);
+
+ // the connection stops answering between two polls: the NOOP of the
next poll times out, and the consumer
+ // connects again
+ server.silenceOpenConnections();
+ await().atMost(10, TimeUnit.SECONDS).until(() ->
server.connectionsOpened() == 2);
+
+ assertTrue(server.commandsOnSilentConnections().contains("NOOP"));
+ // the connection that stopped answering is closed, not left open
behind the new one
+ await().atMost(5, TimeUnit.SECONDS).until(() ->
server.silentConnectionsOpen() == 0);
+ }
+
+ /** Records the name of the files whose processing failed. */
+ static final class RecordingExceptionHandler implements ExceptionHandler {
+
+ final List<String> files = new CopyOnWriteArrayList<>();
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleException(null, null, exception);
+ }
+
+ @Override
+ public void handleException(String message, Throwable exception) {
+ handleException(message, null, exception);
+ }
+
+ @Override
+ public void handleException(String message, Exchange exchange,
Throwable exception) {
+ if (exchange != null) {
+
files.add(exchange.getMessage().getHeader(Exchange.FILE_NAME_ONLY,
String.class));
+ }
+ }
+ }
+}
diff --git
a/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FtpDisconnectFromSilentServerTest.java
b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FtpDisconnectFromSilentServerTest.java
new file mode 100644
index 000000000000..8087f1a142e6
--- /dev/null
+++
b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FtpDisconnectFromSilentServerTest.java
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.file.remote;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.camel.Exchange;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.awaitility.Awaitility.await;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * A disconnect from an FTP server that stopped answering on an open
connection (a hung server, or a connection lost
+ * without being closed) does not wait up to soTimeout for the reply to QUIT:
a stop with nothing in flight returns
+ * within a few seconds, and after a failed poll the forced disconnect does
not send QUIT.
+ */
+public class FtpDisconnectFromSilentServerTest extends CamelTestSupport {
+
+ private static final int SO_TIMEOUT = 15000;
+ // nothing is in flight: the stop should take the shutdown timeout at
most, with a margin, not soTimeout
+ private static final Duration STOP_LIMIT = Duration.ofSeconds(10);
+
+ private FakeFtpServer server;
+
+ @BeforeEach
+ void startServer() throws Exception {
+ server = new FakeFtpServer();
+ context.getShutdownStrategy().setTimeout(2);
+ context.getShutdownStrategy().setTimeUnit(TimeUnit.SECONDS);
+ }
+
+ @AfterEach
+ void stopServer() throws Exception {
+ server.close();
+ }
+
+ private String ftpUri(String directory, int soTimeout, String options) {
+ return "ftp://[email protected]:" + server.port() + "/" + directory +
"?password=secret&passiveMode=true&soTimeout="
+ + soTimeout + "&" + options;
+ }
+
+ @Test
+ void stopIsNotHeldByAnIdleConsumerWhoseServerStoppedAnswering() throws
Exception {
+ server.addFile("/inbox/a.txt",
"hello".getBytes(StandardCharsets.UTF_8));
+ List<String> received = new CopyOnWriteArrayList<>();
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ // one poll at start, then idle: nothing is in flight when the
server stops answering
+ from(ftpUri("inbox", SO_TIMEOUT,
"noop=true&initialDelay=0&delay=600000"))
+ .process(e ->
received.add(e.getMessage().getHeader(Exchange.FILE_NAME, String.class)));
+ }
+ });
+ await().atMost(10, TimeUnit.SECONDS).until(() -> received.size() == 1);
+
+ server.silenceOpenConnections();
+ Duration took = timedStop();
+
+ assertTrue(took.compareTo(STOP_LIMIT) < 0,
+ "CamelContext.stop() took " + took.toMillis() + " ms with
soTimeout " + SO_TIMEOUT + " ms");
+ // the consumer still says goodbye, it does not wait long for the reply
+ assertEquals(List.of("QUIT"), server.commandsOnSilentConnections());
+ }
+
+ @Test
+ void stopIsNotHeldByAnIdleProducerWhoseServerStoppedAnswering() throws
Exception {
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ from("direct:upload").to(ftpUri("outbox", SO_TIMEOUT,
"disconnect=false"));
+ }
+ });
+ template.sendBodyAndHeader("direct:upload", "hello",
Exchange.FILE_NAME, "a.txt");
+ assertTrue(server.files().containsKey("/outbox/a.txt"));
+
+ // the producer keeps its connection (disconnect=false)
+ server.silenceOpenConnections();
+ Duration took = timedStop();
+
+ assertTrue(took.compareTo(STOP_LIMIT) < 0,
+ "CamelContext.stop() took " + took.toMillis() + " ms with
soTimeout " + SO_TIMEOUT + " ms");
+ assertEquals(List.of("QUIT"), server.commandsOnSilentConnections());
+ }
+
+ @Test
+ void failedPollDoesNotSendQuitOnTheConnectionThatStoppedAnswering() throws
Exception {
+ server.addFile("/inbox/a.txt",
"hello".getBytes(StandardCharsets.UTF_8));
+ // the LIST of the first poll gets no reply: the poll fails after
soTimeout, and the poll strategy forces a
+ // disconnect so the next poll connects again
+ server.stallOn(command -> command.startsWith("LIST"),
FakeFtpServer.Stall.BEFORE_REPLY);
+ List<String> received = new CopyOnWriteArrayList<>();
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ from(ftpUri("inbox", 1000,
"noop=true&initialDelay=0&delay=100"))
+ .process(e ->
received.add(e.getMessage().getHeader(Exchange.FILE_NAME, String.class)));
+ }
+ });
+
+ await().atMost(20, TimeUnit.SECONDS).until(() -> received.size() == 1);
+
+ assertEquals(2, server.connectionsOpened());
+ // the forced disconnect closed the connection without sending QUIT
and waiting soTimeout for the reply
+ assertEquals(List.of(), server.commandsOnSilentConnections());
+ await().atMost(5, TimeUnit.SECONDS).until(() ->
server.silentConnectionsOpen() == 0);
+ }
+
+ private Duration timedStop() {
+ long start = System.nanoTime();
+ context.stop();
+ return Duration.ofNanos(System.nanoTime() - start);
+ }
+}
diff --git
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
index d4c1c02bd737..eb619a3b9c95 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
@@ -336,6 +336,27 @@ these messages now reach a started route). The hand-over
on a graceful stop can
the delegated consumer takes to stop (a stop that blocks holds back the
failover until it returns). Exchanges in flight
that the delegated consumer does not wait for in its stop may still be in
flight when the leadership moves, as before.
+=== camel-ftp - a server that stops answering no longer holds the stop and the
rest of the batch
+
+The `ftp` and `ftps` components no longer wait up to `soTimeout` (5 minutes by
default) again and again on a connection
+whose server stopped answering:
+
+* A disconnect, such as when the consumer or the producer stops, still sends
`QUIT`, but waits at most 2 seconds for
+ its reply (`soTimeout` when it is shorter), then closes the connection. The
disconnect that the consumer forces after
+ a failed poll closes the connection without sending `QUIT`.
+* A server that takes more than 2 seconds to answer `QUIT` now makes the
disconnect fail after 2 seconds (the
+ connection is closed all the same). A producer with `disconnect=true` or
`disconnectOnBatchComplete=true` logs it as
+ a WARN (`Exception occurred during disconnecting from: ... This exception is
ignored.`) where it used to wait for the
+ reply; the consumer, and the stop of the consumer or the producer, log it at
DEBUG.
+* When the server does not answer in time while the consumer retrieves a file
(`soTimeout`, or `timeout` during the
+ transfer), only that file fails: the consumer closes the connection and
leaves the other files of the poll to the
+ next poll, which connects again. Before, the consumer went on with the next
file on the same connection, and each
+ remaining file waited `soTimeout` and failed. With `streamDownload=true`,
when the server does not answer the end of
+ a transfer within `soTimeout` (for example after the route timed out reading
the file), the consumer closes the
+ connection and leaves the other files of the poll to the next poll in the
same way.
+
+See the xref:components::ftp-component.adoc[FTP component].
+
=== camel-ftp - sftp checks whether a file exists with one lstat request
The `sftp` component now checks whether a remote file exists
(`fileExist=Ignore|Fail|Move`, `tempFileName`,