shashank created CAMEL-25906:
--------------------------------
Summary: camel-ftp / sftp / smb / azure-files -
checksumFileAlgorithm writes no checksum file since 4.18, or the checksum of an
earlier file producer's payload
Key: CAMEL-25906
URL: https://issues.apache.org/jira/browse/CAMEL-25906
Project: Camel
Issue Type: Bug
Components: camel-file, camel-ftp
Reporter: shashank
Assignee: shashank
{{checksumFileAlgorithm}} ("calculate a checksum from the file that has been
written ... and store the result in the CamelFileChecksum header";
{{checksumWriteFile}} writes it as {{<file>.<algorithm>}}) is declared on
{{GenericFileEndpoint}}, so it is an option of every file component: file, ftp,
ftps, sftp, mina-sftp, scp, smb and azure-files. CAMEL-20172 (4.3) added it for
"camel-file/ftp": {{GenericFileProducer.writeChecksumFile}} calculated the
checksum from the body for every producer.
CAMEL-22846 (4.18) moved the calculation into {{FileOperations.storeFile}} (a
digest while it writes the local file), and
{{GenericFileProducer.writeChecksumFile}} (main 74ee085f7d64, line 290) now
only writes the header that the operations set:
{code:java}
String hash = exchange.getMessage().getHeader(FileConstants.FILE_CHECKSUM,
String.class);
if (hash != null) {
... operations.storeFileDirectly(checksumFileName, hash) ...
}
{code}
Only {{FileOperations}} sets {{CamelFileChecksum}}. So since 4.18:
* an ftp, ftps, sftp, mina-sftp, scp, smb or azure-files producer with
{{checksumFileAlgorithm}} uploads the file and writes no checksum file and no
header, without an error or a log line. A receiver that waits for
{{<file>.sha256}} (or verifies it) never gets it; an upgrade from 4.14 LTS to
4.18 LTS loses the checksum files silently;
* when an earlier file producer of the route set the header, e.g.
{{.to("file:archive?checksumFileAlgorithm=sha256").to("sftp:...?checksumFileAlgorithm=sha256")}},
the remote producer writes that checksum: the checksum of the archive copy,
which is not the uploaded file as soon as the route changed the body in
between, or the remote producer encodes it with its {{charset}}.
*Fix* (in {{GenericFileProducer}}, one place for all components): before the
write, a {{CamelFileChecksum}} header set earlier is removed; for operations
other than {{FileOperations}} the checksum is calculated of the bytes the
remote operations upload. All of them read the body as an {{InputStream}}
({{getMandatoryBody(InputStream.class)}}), or through
{{GenericFileHelper.toInputStream}} when the endpoint has a {{charset}}. A
stream body that can be read once is wrapped in a {{DigestInputStream}} that
the upload reads, so its checksum takes no second pass and no memory; with a
{{charset}} such a stream is kept as a stream cache first (memory, or a spool
file when spooling is enabled). Other bodies (a file, bytes, text, a stream
cache) are read once more for the checksum: for a large file that is a second
read from disk, the price of keeping the fix in one place instead of a digest
in the upload code of each of the seven operations classes. With
{{fileExist=Ignore}} and an existing file nothing is uploaded and no checksum
is written, as with the file component. {{FileOperations}} is unchanged.
Upgrade note in the 4.24 guide.
*Tests*: new {{FtpProducerChecksumFileAlgorithmIT}} (5 tests, camel-ftp, the
embedded FTP server of camel-test-infra-ftp, no Docker): the default
({{checksumWriteFile=true}}) with a String body, a one-shot stream body, a
String and a one-shot stream with {{charset=ISO-8859-1}},
{{checksumWriteFile=false}} (header only), and a file producer earlier in the
route. On main all 5 fail: {{Checksum file should exist ==> expected: <true>
but was: <false>}} (3), {{expected: <a591a6d4...> but was: <null>}} (header),
and {{expected: <01285a4c...> but was: <a591a6d4...>}} (the checksum of the
archived {{Hello World}} written for an uploaded {{Bye World}}). With the fix:
camel-ftp unit tests 81 run, 0 failures (9 skipped); camel-ftp producer ITs
({{*Producer*IT}}: 42 FTP and 4 SFTP classes with the embedded servers) 86 run,
0 failures (3 skipped); camel-core {{org.apache.camel.component.file}} tests
474 run, 0 failures (12 skipped); camel-file 28 run.
*How it was found*: following up the producer paths of
{{checksumFileAlgorithm}} while fixing its charset path in camel-file (separate
ticket): the option is declared on the shared endpoint, the calculation lives
in one subclass of {{GenericFileOperations}}.
*Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (read, not run:
{{writeChecksumFile}} reads only the header, and no remote operations set it).
camel-4.14.x calculated the checksum for every producer.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
