[
https://issues.apache.org/jira/browse/CAMEL-25905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-25905 started by shashank.
----------------------------------------
> camel-file - with checksumFileAlgorithm and charset, the producer decodes the
> body with the JVM default charset (non-ASCII replaced) and writes a checksum
> that does not match the written file
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25905
> URL: https://issues.apache.org/jira/browse/CAMEL-25905
> Project: Camel
> Issue Type: Bug
> Components: camel-file
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> {{FileOperations.storeFile}} (main 74ee085f7d64, lines 349-375) implements
> {{checksumFileAlgorithm}} (CAMEL-22846, "calculate a checksum from the file
> that has been written") by wrapping the body in a {{DigestInputStream}}:
> {code:java}
> if (endpoint.getChecksumFileAlgorithm() != null) {
> MessageDigest digest =
> MessageDigest.getInstance(endpoint.getChecksumFileAlgorithm());
> InputStream is = exchange.getMessage().getBody(FileInputStream.class);
> if (is == null) {
> is = mandatoryBodyStream(exchange, exchange.getMessage(), file);
> }
> dis = new DigestInputStream(is, digest);
> }
> if (charset != null) {
> handleReaderAsFileSource(exchange, file, charset, dis);
> } ...
> {code}
> and {{handleReaderAsFileSource}} (lines 439-456) skips the {{Reader}}
> conversion when it gets that stream and reads it with {{new
> InputStreamReader(is)}}, the default charset of the JVM. So adding
> {{checksumFileAlgorithm}} to a producer with {{charset}} changes what is
> written, and the checksum is that of the body, not of the file:
> * the bytes of a file consumed with {{charset=ISO-8859-1}} (or any body whose
> bytes are in the exchange charset) are decoded as UTF-8: every non-ASCII
> character is written as U+FFFD. Without the checksum option the same route
> writes the file right
> ({{from("file:in?charset=ISO-8859-1").to("file:out?charset=UTF-8&checksumFileAlgorithm=sha256")}}
> turns {{Grüße aus Köln}} into {{Gr??e aus K?ln}});
> * the checksum (the {{CamelFileChecksum}} header and the {{.sha256}} file) is
> calculated over the bytes before re-encoding, so it does not match the
> written file as soon as the encoding changes a byte: a receiver that verifies
> the checksum file rejects every such file;
> * a String body whose exchange carries the endpoint charset (for example sent
> with a {{ProducerTemplate}} to
> {{file:out?charset=ISO-8859-1&checksumFileAlgorithm=sha256}}) is converted to
> ISO-8859-1 bytes, decoded as UTF-8 and fails with
> {{UnmappableCharacterException}};
> * with {{fileExist=Append}} and {{appendChars}}, the appended characters are
> written after the digest stream was read, so the checksum leaves them out.
> *Fix*: the digest is updated with the bytes written to the file: a
> {{DigestOutputStream}} under the writer of the charset path, and the buffer
> and the {{appendChars}} bytes of the stream path. The body is decoded as
> without the checksum option (the {{Reader}} conversion with the exchange
> charset). The writer of the charset path still reports unmappable characters,
> as {{Files.newBufferedWriter}} did. Upgrade note in the 4.24 guide (checksums
> of such files differ from the ones written before).
> *Tests*: new {{FileProducerChecksumOfWrittenFileTest}} in camel-core (4
> tests). On main: {{transcodedFileHasItsOwnChecksumAndContent}} fails with
> {{array lengths differ, expected: <17> but was: <20>}} (three U+FFFD of three
> bytes each), {{stringWrittenWithCharsetHasTheChecksumOfTheFile}} fails with
> {{CamelExecutionException}} caused by {{UnmappableCharacterException}},
> {{appendCharsArePartOfTheChecksum}} fails with {{expected: <d2a84f4b...> but
> was: <a591a6d4...>}} (the checksum of {{Hello World}} without the newline).
> {{bodyWithoutCharsetKeepsItsChecksum}} is a control and passes on main. The 9
> tests of {{FileProducerChecksumFileAlgorithmTest}} pass with the fix. The
> tests of the {{org.apache.camel.component.file}} package of camel-core: 478
> run, 0 failures, 12 skipped; camel-file: 28 run, 0 failures (one flake of
> {{FileProducerNoForcedWritesTest}}, passed on rerun; it does not use the
> checksum option).
> *How it was found*: comparing the code paths of the file producer for one
> option (here: the checksum path replaces the input of the charset path). The
> JUnit tests above are the evidence.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x ({{new
> DigestInputStream}} and {{new InputStreamReader(is)}} at the same lines;
> read, not run). 4.14.x has an older implementation
> ({{GenericFileProducer.writeChecksumFile}} re-reads the body after the
> write): no decoding problem there, but the same checksum of the body instead
> of the file.
> Not changed: with {{fileExist=Append}} to an existing file, the checksum
> covers the bytes this exchange wrote, not the whole file. The remote file
> producers (ftp, sftp, scp, smb, azure-files) expose {{checksumFileAlgorithm}}
> too (inherited from {{GenericFileEndpoint}}) but since the checksum moved
> into {{FileOperations}} (CAMEL-22846, 4.18) they never calculate one, so no
> header and no checksum file are written (4.14.x wrote one for them); possible
> follow-up.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
