Bruno Gonçalves created CAMEL-27264:
---------------------------------------
Summary: camel-ftp - files polled with pollEnrich from sftp stay
in the in-progress repository and are skipped until restart when the enriched
exchange completes on another thread
Key: CAMEL-27264
URL: https://issues.apache.org/jira/browse/CAMEL-27264
Project: Camel
Issue Type: Bug
Components: camel-ftp
Affects Versions: 4.23.0, 4.10.9
Reporter: Bruno Gonçalves
Attachments: sftp-pollenrich-stuck-inprogress-reproducer.tar.gz
h3. Problem
A route polls an sftp folder with {{pollEnrich}} (one file per call, {{move}}
to an archive folder) and then sends the exchange with JMS request-reply
({{InOut}}, {{requestTimeout=20000}}) to a second route. After a while some
files are no longer picked up: they stay in the folder, are never processed,
and every later poll skips them. Nothing is logged above TRACE ("Skipping as
file is already in progress"). Only a JVM restart, or renaming the file, makes
them visible again.
Their names stay in the endpoint's in-progress repository
({{GenericFileEndpoint.getInProgressRepository()}}, a
{{MemoryIdempotentRepository}}). Because the {{pollEnrich}} endpoint is
resolved dynamically, it stays in the endpoint registry when the route is
removed, so stopping or reinstalling the route does not clear the repository
either.
h3. Consumer options
{noformat}
sftp:host/data/outbound/inbox?autoCreate=false&delete=false&recursive=false&passiveMode=true
&readLock=changed&fastExistsCheck=true&disconnect=true&throwExceptionOnConnectFailed=true
&bridgeErrorHandler=true&binary=true&move=archive&sortBy=file:modified
&maxMessagesPerPoll=1&eagerMaxMessagesPerPoll=false {noformat}
{{pollEnrich}} uses {{timeout=5000}} and an aggregation strategy that copies
the polled body and headers onto the original exchange.
h3. What we see
At the same time as the files get stuck, the sftp operations of the endpoint
fail with errors like these (Camel 4.23.0):
{noformat} org.apache.camel.component.file.GenericFileOperationFailedException:
Cannot list directory: data/outbound/inbox/ProductChanges_...offset_300....xml
at
org.apache.camel.component.file.remote.SftpOperations.listFiles(SftpOperations.java:936)
at
org.apache.camel.component.file.remote.strategy.SftpExclusiveReadLockCheck.listFilesFast(SftpExclusiveReadLockCheck.java:120)
at
org.apache.camel.component.file.remote.strategy.SftpExclusiveReadLockCheck.tryAcquireExclusiveReadLock(SftpExclusiveReadLockCheck.java:55)
at
org.apache.camel.component.file.remote.strategy.SftpChangedExclusiveReadLockStrategy.acquireExclusiveReadLock(SftpChangedExclusiveReadLockStrategy.java:73)
at
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:430)
at
org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(RemoteFileConsumer.java:197)
at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:216)
at
org.apache.camel.component.file.GenericFilePollingConsumer.doReceive(GenericFilePollingConsumer.java:152)
at
org.apache.camel.component.file.GenericFilePollingConsumer.receive(GenericFilePollingConsumer.java:102)
Caused by: 4: java.io.IOException: Pipe closed
org.apache.camel.CamelExchangeException: Error during poll. Caused by:
[GenericFileOperationFailedException - Cannot change directory to: ..] at
org.apache.camel.processor.PollEnricher.process(PollEnricher.java:379) Caused
by: org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
change directory to: .. at
org.apache.camel.component.file.remote.SftpOperations.doChangeDirectory(SftpOperations.java:887)
at
org.apache.camel.component.file.remote.SftpOperations.changeCurrentDirectory(SftpOperations.java:872)
at
org.apache.camel.component.file.remote.AbstractSftpConsumer.pollDirectory(AbstractSftpConsumer.java:94)
at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:151)
at
org.apache.camel.component.file.GenericFilePollingConsumer.doReceive(GenericFilePollingConsumer.java:152)
at
org.apache.camel.component.file.GenericFilePollingConsumer.receive(GenericFilePollingConsumer.java:102)
Caused by: 4: java.io.IOException: Pipe closed at
com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:344) {noformat}
In production (4.10.9) we also see the move to the archive folder fail
({{Cannot rename file ... 2: No such file}}, {{Cannot create directory:
.../archive}}) on the JMS reply thread, within a second of a poll failing on
the route's own thread.
h3. Analysis so far
For file endpoints {{pollEnrich}} uses {{GenericFilePollingConsumer}}, which
polls synchronously on the calling thread ({{doReceive}} ->
{{GenericFileConsumer.poll()}}).
With JMS request-reply after the {{pollEnrich}}, the exchange (and the
handed-over {{GenericFileOnCompletion}} that moves the file and removes the
name from the in-progress repository) completes on the JMS reply-manager thread
({{TemporaryQueueReplyManager}} or {{JmsReplyManagerOnTimeout}}).
So the commit of file N runs on the reply thread while the poll for file N+1
runs on the route's thread, both through the endpoint's single
{{SftpOperations}}. With {{disconnect=true}} one side closes the channel under
the other ("Pipe closed"). With the default {{stepwise=true}} one side changes
the current directory under the other ("Cannot change directory to: ..", "No
such file" on the rename).
Some of these collisions leave the file's name in the in-progress repository.
We have not pinned down the exact code path. In one run a stuck file was moved
to the archive folder much later, so at least some entries are released
eventually rather than lost. In other runs a renamed copy of a stuck file was
not picked up within 2 minutes either.
h3. Reproducer
Attached: {{sftp-pollenrich-stuck-inprogress-reproducer.tar.gz}} (plain Java,
JDK 21, about 270 lines). It starts an embedded sftp server (Apache MINA SSHD,
40 ms added per request) and an embedded ActiveMQ broker, and runs:
a writer route that writes 5 files into the inbox every 60 s,
a timer route (every 10 s, runs never overlap) with the {{pollEnrich}} above,
followed by
{{to("activemq:queue:next-route?exchangePattern=InOut&requestTimeout=20000")}},
a second route on that queue that usually answers within 5 s and in one of five
calls takes 15-30 s (so some requests time out).
It reads the endpoint's in-progress repository every second and, after the
writer stops and the poller has drained the folder, reports files that were
never processed and whose name has been in the repository for over 60 s.
{noformat} $ mvn compile exec:java -Dexec.args="current 600 600"
written=50 processed=40 left in inbox=14 stuck (never processed, in progress
for over 60 s)=10 STUCK
ProductChanges_20261119_180332_offset_0_20261119T180332237.xml ...
$ mvn compile exec:java -Dexec.args="fixed 600 600"
written=50 processed=50 left in inbox=0 stuck (never processed, in progress for
over 60 s)=0 {noformat}
Use {{-Dcamel.version=4.23.0 -Dactivemq.version=6.3.2}} for the latest release
(default 4.10.9 with 6.1.5).
||Camel||Options||Written||Stuck||SFTP errors||
|4.10.9|as above|50|12|34x Pipe closed, 6x Cannot change directory|
|4.23.0|as above|50|10|32x Pipe closed, 2x Cannot change directory to: ..|
|4.23.0|fixed|50|0|none (18 request timeouts)|
|4.10.9|as above, seda instead of JMS request-reply|50|0|none|
"fixed" adds {{readLockCheckInterval=1000&readLockTimeout=4000&stepwise=false}}
and uses {{disconnect=false}}. We have not yet tested these options one by one.
h3. Expected
A file whose exchange has finished, failed or timed out should always be
removed from the in-progress repository, as for the poll-error case in
CAMEL-7407. If an entry can stay for a long time, a WARN log would make the
silent skipping visible.
h3. Related
CAMEL-7407: FTP route considers files as in progress after error in poll (fixed
in 2.12.4, 2.13.2, 2.14.0).
----
_This ticket was created using Claude (Anthropic). The reproducer and results
were run and checked by the reporter._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)