allthingssecurity opened a new pull request, #28430: URL: https://github.com/apache/camel/pull/28430
# Description [CAMEL-25906](https://issues.apache.org/jira/browse/CAMEL-25906) `checksumFileAlgorithm` is declared on `GenericFileEndpoint`, so ftp, ftps, sftp, mina-sftp, scp, smb and azure-files all expose it, and CAMEL-20172 added it for "camel-file/ftp". Since CAMEL-22846 (4.18) the checksum is calculated in `FileOperations` while it writes, and `GenericFileProducer.writeChecksumFile` only writes the `CamelFileChecksum` header that the operations set. The remote operations never set it, so: - the remote producers upload the file and write no checksum file and no header, without an error (4.14 wrote one); - after an earlier file producer in the route, e.g. `.to("file:archive?checksumFileAlgorithm=sha256").to("sftp:...?checksumFileAlgorithm=sha256")`, the remote producer writes the archive's checksum, which is wrong as soon as the body changed in between or the remote producer encodes it with its `charset`. Changes (`GenericFileProducer` only, so every file component gets the same behaviour): - Before the write, a `CamelFileChecksum` header set earlier in the route is removed. - For operations other than `FileOperations`, the checksum is calculated of the bytes the remote operations upload. ftp, ftps, sftp, mina-sftp and smb read the body with `getMandatoryBody(InputStream.class)`, or with `GenericFileHelper.toInputStream` when the endpoint has a `charset`. scp (`ScpOperations`) and azure-files (`FilesOperations`) ignore `charset` and upload the bytes of the body as they are; with `charset` set there the checksum (of the encoded bytes) does not match the uploaded file, which the upgrade note says (do not combine `charset` and `checksumFileAlgorithm` with them). - A body that gives the same bytes each time it is read (text, bytes, a `File`/`Path`, a `WrappedFile` of a local file or whose body is one of these, a stream cache) is read once more for the checksum. Any other body is turned into a stream cache first when a converter exists (a `Reader` becomes a `ReaderCache`), so the checksum does not use it up before the upload (found in review: a `Reader` body failed the upload with `Stream closed`). An `InputStream` that can be read once (or a body without a stream cache converter) is wrapped in a `DigestInputStream` that the upload reads, so its checksum takes no second pass and no memory; with a `charset` such a stream is kept as a stream cache first. azure-files sizes the remote file from the body length; the `DigestInputStream` hides it, so azure-files then reads such a stream into memory (with its existing WARN). - The `CamelFileChecksum` header is set once the file is in place (after the upload and the rename of a temporary file), so an upload that fails (and is handled) or a rename skipped with `fileExist=Ignore` leaves no header. - With `fileExist=Ignore` and an existing file, nothing is uploaded and no checksum is written, as with the file component (one extra `existsFile` call, only in that mode with a checksum). - `FileOperations` is unchanged; upgrade note in the 4.24 guide (it also notes that FTP uploads in ASCII mode unless `binary=true`, where the server may change line ends, so the stored file can differ from the checksummed bytes, as in 4.14). Cost: with `checksumFileAlgorithm`, a file, byte or text body is read once more for the checksum (for a large file, a second read from disk); a one-shot stream with a `charset` is stream-cached (memory, or a spool file when spooling is enabled). A digest inside the upload of each operations class (ftp, sftp, mina-sftp, scp, smb, azure-files) would avoid the second read, at the price of seven copies of the same code; I kept it in one place. Happy to move it into the operations if you prefer. Tests: - New `FtpProducerChecksumFileAlgorithmIT` (7 tests, the embedded FTP server of camel-test-infra-ftp, no Docker): the default (`checksumWriteFile=true`) with a String body and with a one-shot stream, a String and a one-shot stream with `charset=ISO-8859-1`, `checksumWriteFile=false` (header only), and a file producer earlier in the route. On main all 5 fail: `Checksum file should exist ==> expected: <true> but was: <false>` (3), `expected: <a591a6d4...> but was: <null>` (header), and `expected: <01285a4c...> but was: <a591a6d4...>` (the archived `Hello World` checksum for an uploaded `Bye World`). Two tests from review fail on the previous version of this branch (747ffabc68): `testChecksumOfReaderBody` (a `StringReader` body, with and without `charset`) with `CamelExecutionException ... Stream closed`, and `testNoChecksumHeaderWhenTheExistingFileIsKept` (`tempFileName`, `fileExist=Ignore`, `eagerDeleteTargetFile=false`, existing target) with `expected: <null> but was: <01285a4c... >`. - camel-ftp unit tests and FTP producer ITs (`*Test` and `FtpProducer*IT`, embedded server): 156 run, 0 failures (12 skipped). camel-file: 28 run, 0 failures. (Before the review fixes: camel-ftp unit 81, producer ITs 86, camel-core `org.apache.camel.component.file` 474, all 0 failures; the review fixes touch only the non-`FileOperations` path.) What was not run: the smb, azure-files, scp and mina-sftp tests (the change is in the shared producer; they go through the same code); the full root build. This PR and the camel-file charset/checksum fix (CAMEL-25906, branch camel-file-checksum-of-written-bytes) touch different files and merge cleanly in either order. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-25906](https://issues.apache.org/jira/browse/CAMEL-25906). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built camel-file, camel-ftp and camel-core with their dependencies and ran the tests above, including the formatter and import-sort plugins; no generated file changes. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
