Croway opened a new pull request, #29246: URL: https://github.com/apache/camel/pull/29246
## Summary [CAMEL-26635](https://issues.apache.org/jira/browse/CAMEL-26635): with `maxMessagesPerPoll`, the file consumers listed the whole directory on every poll, and only then stopped picking files. A user consuming a folder of 25k+ files from Azure Blob Storage SFTP with `maxMessagesPerPoll=1000` hit this. The `eagerMaxMessagesPerPoll` description already said "If eager then the limit is during the scanning of files". With the eager limit (the default), the file, sftp, mina-sftp and smb consumers now read the directory one entry at a time, and stop once the poll has its files. The stop counts **accepted** files (`canPollMoreFiles`, after include/exclude, filters, idempotent and in-progress checks), not raw entries. A raw cap would starve when rejected entries stay in the directory, for example with `noop`. There is no new option. The whole directory is still listed when something needs it: - `preSort` sorts all its files; - `doneFileName` looks the done file up among them; - `eagerMaxMessagesPerPoll=false`. Each poll still lists from the start, as before: - with `delete` or `move`, a poll reads about `maxMessagesPerPoll` entries; - with `noop` or idempotent, it reads further on each poll. ## Changes - **camel-file**: - `GenericFileListingOperations<T>`: operations that list a directory one entry at a time, with `isReentrantListing()`; - `GenericFileConsumer.canStopListingAtLimit()`; - `FileConsumer` reads the directory with a `DirectoryStream` (an unreadable directory still has no files, as with `File.listFiles()`). - **camel-ftp-common**: `AbstractSftpConsumer` polls from the lazy listing. The loop body is extracted, unchanged, into `pollFile`. - **camel-mina-sftp**: `openDir` and `listDir(handle)`. `readDir(path)` only closes the handle at the end of the directory, so it would leak when the listing stops early. - **camel-sftp (JSch)**: `ls` with an `LsEntrySelector`. - JSch calls the selector while it is still reading a reply, in a buffer shared by every request, so no request can be sent from it. A `stat` in the selector makes `ls` fail with `ArrayIndexOutOfBoundsException`. - The listing therefore stops at the first entry that needs a request: a sub directory to poll with `recursive`, or a symbolic link to resolve with `jailStartingDirectory`. - The rest of that directory is then polled from the whole listing, as before, skipping the entries already polled. Flat directories are read lazily all the way. - **camel-smb**: `Directory.iterator`, through a new `SmbOperations.listFiles(path, searchPattern, visitor)`. - The exception is a server that supports SMB 3 directory leases (Samba, Windows) with no `searchPattern`. smbj 0.15 then serves a repeat listing of the whole directory from its cache, which is cheaper than reading the start of the directory again. The lazy listing was 7× slower on repeat polls there, so the whole listing is kept. - `SmbOperations` does not implement the new interface: `listFiles(String, Predicate)` would make the existing `listFiles(path, null)` ambiguous. - **Docs**: - the `eagerMaxMessagesPerPoll` description; - a "Polling large directories" section in the smb docs; - a "TCP_NODELAY" section in the mina-sftp docs. Apache MINA SSHD does not set `TCP_NODELAY`, while JSch does. `-Dorg.apache.sshd.config.tcp-nodelay=true` takes a poll of 10,000 files from 66 ms to 18 ms. - Generated metadata, the catalog and the endpoint DSL are regenerated. ## Performance Measured with the real consumers, before (main 1164a17977) and after this change, with `maxMessagesPerPoll=1000`. Servers, all in Docker (OrbStack, arm64): - OpenSSH `sftp-server`, which sends at most 100 entries per READDIR reply; - Samba (Fedora 43, as `camel-test-infra-smb`); - kernel NFS server, NFSv4.2; - toxiproxy, adding 10 ms in each direction (about 20 ms round trip). ### One poll of a directory that keeps its files `noop=true&idempotent=false`, so every poll reads the same directory; `download=false`. Each row is the median poll time, except the NFS first-poll rows. | Consumer | Network | Files | Before | After | |---|---|---:|---:|---:| | file | local disk | 10,000 | 15 ms | 12 ms | | file | local disk | 50,000 | 21 ms | 12 ms | | file | NFS | 10,000 | 15 ms | 15 ms | | file | NFS | 50,000 | 22 ms | 14 ms | | file, first poll | NFS, 20 ms round trip | 10,000 | 71.4 s | 0.38 s | | file, first poll | NFS, 20 ms round trip | 50,000 | 71.5 s | 0.49 s | | smb, server with directory leases | direct | 50,000 | 7 ms | 5 ms | | smb, server with directory leases | 20 ms round trip | 50,000 | 6 ms | 5 ms | | smb, server without directory leases | direct | 50,000 | 241 ms | 42 ms | | smb, server without directory leases | 20 ms round trip | 50,000 | 493 ms | 112 ms | | sftp | direct | 10,000 | 59 ms | 16 ms | | sftp | direct | 50,000 | 186 ms | 14 ms | | sftp | 20 ms round trip | 10,000 | 2.9 s | 385 ms | | sftp | 20 ms round trip | 50,000 | 14.2 s | 376 ms | | mina-sftp | direct | 10,000 | 760 ms | 67 ms | | mina-sftp | direct | 50,000 | 3.7 s | 64 ms | | mina-sftp | 20 ms round trip | 10,000 | 3.6 s | 468 ms | | mina-sftp | 20 ms round trip | 50,000 | 17.2 s | 469 ms | Memory allocated on the consumer thread per poll also drops, for example: - sftp, 50,000 files: 25 MB to 6 MB; - smb without directory leases: 30 MB to 5 MB; - file on local disk: 12 MB to 3 MB. The first NFS poll gains more than the listing itself costs: - After `File.listFiles()`, each file Camel checks costs an NFS `LOOKUP` round trip: about 1,000 for the 1,000 files of the poll. - When the directory is read while the files are checked, the Linux client takes the attributes from the `READDIR` replies: 18 `LOOKUP` and 7 `READDIR`. - Counted from `/proc/self/mountstats` with a cold cache. ### Consuming a whole directory `delete=true&greedy=true`. Each row is the total time to consume the directory. | Consumer | Network | Files | Before | After | |---|---|---:|---:|---:| | file | local disk | 20,000 | 0.5 s | 0.5 s | | file | NFS | 20,000 | 2.9 s | 2.1 s | | smb, server with directory leases | direct | 20,000 | 9.1 s | 9.1 s | | smb, server without directory leases | direct | 20,000 | 9.0 s | 8.7 s | | sftp | direct | 20,000 | 3.1 s | 2.3 s | | mina-sftp | direct | 20,000 | 20.1 s | 5.7 s | Before, every poll re-read what was left of the directory: 20k + 19k + … + 1k entries to consume 20k files. ## Tests - `MinaSftpConsumerMaxMessagesPerPollListingIT` and `SftpConsumerMaxMessagesPerPollListingIT`: - The embedded server counts the directory entries it sends (`SftpEventListener`). - **Stop:** a poll of 10 files from a directory of 1,000 reads fewer than 1,000 entries, with `stepwise` true and false. - **Fallbacks:** `preSort`, `doneFileName` and `eagerMaxMessagesPerPoll=false` still read all 1,002. - **No missed files:** every file is consumed over the polls with `noop`, and with `recursive`. - **Symbolic links:** those inside the starting directory are consumed, the others skipped. - **No leaks:** every directory opened is closed. - The sftp IT also covers many files listed after a sub directory (the JSch fallback). - `SmbConsumerMaxMessagesPerPollListingIT`: with and without directory leases, with `recursive`, and with `searchPattern`. - `FileConsumeMaxMessagesPerPollListingTest`: `noop` over the polls, `recursive`, hidden directories, an unreadable directory, and the `preSort` and done file fallbacks. - **Control on main:** only the two "stops listing at the limit" tests fail (1,002 entries read). - **Mutation:** with JSch marked as reentrant, the four recursion and symbolic link tests fail. - **Speed:** the new tests take about 15 s together, without the Samba container start. - **Also run:** the file tests of camel-core, and the unit tests and SFTP, MINA SFTP and SMB integration tests of the changed modules. _Claude Code on behalf of Federico Mariani_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
