shashank created CAMEL-25904:
--------------------------------
Summary: camel-beanio - BeanIOSplitter reads a file with the
default charset of the JVM and ignores its encoding, so a file consumed with
charset=ISO-8859-1 is split with its non-ASCII characters replaced
Key: CAMEL-25904
URL: https://issues.apache.org/jira/browse/CAMEL-25904
Project: Camel
Issue Type: Bug
Components: camel-beanio
Reporter: shashank
Assignee: shashank
{{BeanIOSplitter.evaluate}} (main 74ee085f7d64, lines 79-107) creates the
BeanIO reader like this:
{code:java}
if (body instanceof WrappedFile wrappedfile) {
body = wrappedfile.getFile();
}
if (body instanceof File file) {
beanReader = factory.createReader(getStreamName(), file);
}
if (beanReader == null) {
Reader reader = msg.getMandatoryBody(Reader.class);
beanReader = factory.createReader(getStreamName(), reader);
}
{code}
BeanIO's {{StreamFactory.createReader(String, File)}} (beanio 3.2.1) opens the
file with a {{FileReader}}, that is with the default charset of the JVM. Other
bodies (byte array, stream) are converted to a {{Reader}} with the charset of
the exchange, else the default charset. The {{encoding}} option of the splitter
({{setEncoding}}, kept in its {{BeanIOConfiguration}}) is never read.
The usual route for big files is a file consumer and the streaming splitter,
and the file body is always a {{File}}:
{code:java}
from("file:in?charset=ISO-8859-1")
.split(beanIOSplitter).streaming()
.to("direct:record");
{code}
The {{charset}} of the file consumer is ignored, and so is
{{beanIOSplitter.setEncoding(StandardCharsets.ISO_8859_1)}}: an ISO-8859-1 or
windows-1252 file (the typical encoding of the legacy fixed-length and CSV
files BeanIO is used for) is decoded as UTF-8 on Java 18+, and every non-ASCII
character of the records becomes U+FFFD ({{Jörg Müller}} -> {{J?rg M?ller}}).
The split completes without an error. CAMEL-12284 fixed the same for
{{BeanIODataFormat}} (its encoding option did not work); the splitter was added
later (CAMEL-9843) and has never used it.
*Fix*: {{BeanIOSplitter}} decodes a file, stream or byte array body itself,
with the charset of the exchange ({{CamelCharsetName}} header or property, as
set by the {{charset}} option of the file consumer), else with its {{encoding}}
(default: the default charset of the JVM, as before). A {{String}} or
{{Reader}} body is read as before. The reader is closed if BeanIO cannot create
its reader (an unknown stream name). Upgrade note in the 4.24 guide.
*Tests*: new {{BeanIOSplitterEncodingTest}} (4 tests, ISO-8859-1 data with ö,
ü, é, ß). On main, 3 fail with {{expected: <[Jörg Müller Développeur, Renée
Straße Architect]> but was: <[J?rg M?ller D?veloppeur, Ren?e Stra?e
Architect]>}} (U+FFFD shown as ?): {{fileIsReadWithTheEncodingOfTheSplitter}}
(a {{File}} body, {{setEncoding(ISO_8859_1)}}),
{{fileFromTheFileConsumerIsReadWithItsCharset}} (a real file consumer with
{{charset=ISO-8859-1}}), {{bytesAreReadWithTheEncodingOfTheSplitter}}.
{{bytesAreReadWithTheCharsetOfTheExchange}} passes on main and with the fix
(control: that path already used the exchange charset). camel-beanio unit
tests: 25 run, 0 failures.
*How it was found*: a sweep of the data formats and their streaming splitters
for options that one code path does not use (here the {{File}} branch hands the
file to a library method without a charset). The JUnit tests above are the
evidence.
*Affected*: main, camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x
(the same {{createReader(getStreamName(), file)}}; read, not run).
Not changed, possible follow-up: {{BeanIOIterator}} reads the first record in
its constructor, before {{evaluate}} sets the error handler on the reader, so
the {{ignoreInvalidRecords}} / {{ignoreUnexpectedRecords}} / custom
{{BeanReaderErrorHandler}} options do not apply to the first record (an invalid
first record fails the split, loudly).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
