davsclaus opened a new pull request, #3178:
URL: https://github.com/apache/camel-kamelets/pull/3178
Part of #3170: `ftp-source`, `ftps-source` and `sftp-source` can move the
files they read, so that each file is read only once, also across restarts.
### The problem
Without `delete`, these sources read every file on the server again after a
restart. The list of files already read (`idempotent`) is kept only in memory.
### The fix: `move` and `moveFailed`
| Option | What it does |
|---|---|
| `move` | Where to move each file on the server once it has been read and
handed on, such as `.done`. A plain directory name is relative to the directory
of the file. A path that ends with the file name, such as
`done/${date:now:yyyyMMdd}/${file:name}`, is relative to `directoryName`. |
| `moveFailed` | Where to move a file when the next step fails, such as
`.error`. Without it, a failing file is read again at every poll. |
- **Defaults:** both options are optional. With them unset, the endpoint URI
is byte-identical to today's.
- **Descriptions:**
- Dot-directories such as `.done` are never read, which matters with
`recursive`.
- The server user needs the right to create directories and rename files.
- `move` together with `delete` stops the integration from starting.
- **Pointers:** the "Good to know" paragraph and the `idempotent` and
`delete` descriptions point to `move`.
- **Partial pages:** each has an example.
### Why not a persistent idempotent repository
A persistent idempotent repository was the first idea, but `move` is better
here:
- **Less to set up:** `move` needs no Camel knowledge.
- **Survives redeploys:** the state lives on the server, so it survives pod
restarts and redeploys. A local file store is lost without a persistent volume.
- **Not possible from a Kamelet option:** a `FileIdempotentRepository`
cannot be created only when a file path is given. Without a path it fails to
start instead of falling back to memory. A bean the user declares cannot set
`fileStore` from a string either, because Camel has no String to File type
converter.
A follow-up in `apache/camel`, such as a String to File converter, would
make an `idempotentRepository` option possible later.
### Verification
Tested with Camel JBang against `atmoz/sftp`, against
`delfer/alpine-ftp-server` for FTP, and against the same image with TLS for
FTPS. Each route ran twice, to simulate a restart.
| Case | Result |
|---|---|
| Options unset | The second run read the same files again, as today |
| `move: .done` and `moveFailed: .error`, with a step that fails for `bad*`
files | Run 1 moved `a` and `b` to `.done/` and `bad` to `.error/`. Run 2 read
only the new file `c.txt`. This held on FTP, FTPS and SFTP |
| `recursive: true` with `.done` | `2026/f.txt` moved to `2026/.done/` and
was not read again |
| `move: done/${date:now:yyyyMMdd}/${file:name}` | Moved to
`upload/done/20261117/h.txt` |
| `move` with `delete: true` | The integration fails to start, as the
description says |
| A failing file without `moveFailed` | Read again at every poll |
`./mvnw clean install` passed twice, the first time with `-DskipTests`.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]