[
https://issues.apache.org/jira/browse/CAMEL-25904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-25904 started by shashank.
----------------------------------------
> camel-beanio - BeanIOSplitter reads a file with the default charset of the
> JVM and ignores its encoding, so a file consumed with charset=ISO-8859-1 is
> split with its non-ASCII characters replaced
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25904
> URL: https://issues.apache.org/jira/browse/CAMEL-25904
> Project: Camel
> Issue Type: Bug
> Components: camel-beanio
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> {{BeanIOSplitter.evaluate}} (main 74ee085f7d64, lines 79-107) creates the
> BeanIO reader like this:
> {code:java}
> if (body instanceof WrappedFile wrappedfile) {
> body = wrappedfile.getFile();
> }
> if (body instanceof File file) {
> beanReader = factory.createReader(getStreamName(), file);
> }
> if (beanReader == null) {
> Reader reader = msg.getMandatoryBody(Reader.class);
> beanReader = factory.createReader(getStreamName(), reader);
> }
> {code}
> BeanIO's {{StreamFactory.createReader(String, File)}} (beanio 3.2.1) opens
> the file with a {{FileReader}}, that is with the default charset of the JVM.
> Other bodies (byte array, stream) are converted to a {{Reader}} with the
> charset of the exchange, else the default charset. The {{encoding}} option of
> the splitter ({{setEncoding}}, kept in its {{BeanIOConfiguration}}) is never
> read.
> The usual route for big files is a file consumer and the streaming splitter,
> and the file body is always a {{File}}:
> {code:java}
> from("file:in?charset=ISO-8859-1")
> .split(beanIOSplitter).streaming()
> .to("direct:record");
> {code}
> The {{charset}} of the file consumer is ignored, and so is
> {{beanIOSplitter.setEncoding(StandardCharsets.ISO_8859_1)}}: an ISO-8859-1 or
> windows-1252 file (the typical encoding of the legacy fixed-length and CSV
> files BeanIO is used for) is decoded as UTF-8 on Java 18+, and every
> non-ASCII character of the records becomes U+FFFD ({{Jörg Müller}} -> {{J?rg
> M?ller}}). The split completes without an error. CAMEL-12284 fixed the same
> for {{BeanIODataFormat}} (its encoding option did not work); the splitter was
> added later (CAMEL-9843) and has never used it.
> *Fix*: {{BeanIOSplitter}} decodes a file, stream or byte array body itself,
> with the charset of the exchange ({{CamelCharsetName}} header or property, as
> set by the {{charset}} option of the file consumer), else with its
> {{encoding}} (default: the default charset of the JVM, as before). A
> {{String}} or {{Reader}} body is read as before. The reader is closed if
> BeanIO cannot create its reader (an unknown stream name). Upgrade note in the
> 4.24 guide.
> *Tests*: new {{BeanIOSplitterEncodingTest}} (4 tests, ISO-8859-1 data with ö,
> ü, é, ß). On main, 3 fail with {{expected: <[Jörg Müller Développeur, Renée
> Straße Architect]> but was: <[J?rg M?ller D?veloppeur, Ren?e Stra?e
> Architect]>}} (U+FFFD shown as ?): {{fileIsReadWithTheEncodingOfTheSplitter}}
> (a {{File}} body, {{setEncoding(ISO_8859_1)}}),
> {{fileFromTheFileConsumerIsReadWithItsCharset}} (a real file consumer with
> {{charset=ISO-8859-1}}), {{bytesAreReadWithTheEncodingOfTheSplitter}}.
> {{bytesAreReadWithTheCharsetOfTheExchange}} passes on main and with the fix
> (control: that path already used the exchange charset). camel-beanio unit
> tests: 25 run, 0 failures.
> *How it was found*: a sweep of the data formats and their streaming splitters
> for options that one code path does not use (here the {{File}} branch hands
> the file to a library method without a charset). The JUnit tests above are
> the evidence.
> *Affected*: main, camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x
> (the same {{createReader(getStreamName(), file)}}; read, not run).
> Not changed, possible follow-up: {{BeanIOIterator}} reads the first record in
> its constructor, before {{evaluate}} sets the error handler on the reader, so
> the {{ignoreInvalidRecords}} / {{ignoreUnexpectedRecords}} / custom
> {{BeanReaderErrorHandler}} options do not apply to the first record (an
> invalid first record fails the split, loudly).
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
