Hello again.
Thanks a lot for your feedback and the links to the commits. Indeed,
following your way, we've checked the source code from Red-Hat and the
camel-core 2.17.0.redhat-630516 we use has a different code, that is
not a backport of the Apache's commit, but leads to the same issue.
The code uses a FileOutputStream like the original code, but the
charset is set with a different way that probably produces the same
exception.
FileOutputStream os = new FileOutputStream(target, append);
Writer out = IOConverter.toWriter(os,
Charset.forName(charset).newEncoder());
This explains why the "original" 2.17.0 works on my machine and fails
on Fuse (that provides a divergent version).
And your explanations seem to confirm why the 2.25.4 also fails, but
for another reason.
It seems that the only way to manage the issue is to code a workaround
as Raymond said in another mail, to avoid processing "non ISO-8859-1"
characters on the file producer that must use an explicit charset.
Something like that (which is not a really pretty coding) seems to work :
from("file:./input?delete=true&charset=UTF-8&moveFailed=.BUG") //
.log("PMA : processing ${file:name} (length ${file:length}) : ${body}") //
.convertBodyTo(byte[].class, "ISO-8859-1") //
.convertBodyTo(String.class, "ISO-8859-1") //
.log("PMA : after conversion : ${body}") //
.to("file:./output?charset=ISO-8859-1");
In our case, the body should be rather small so as not to consume too
much memory. Not sure this could be a suitable solution for other
cases with bigger bodies...
Thanks a lot for your help !!!
I didn't think before that a trivial route with files and charsets
could be so hard to code :(
Regards.
Le jeu. 3 nov. 2022 à 17:07, Zheng Feng <zf...@redhat.com> a écrit :
>
> I just checked the commits history. And it seems related to this change [1]
> which switched to using Files.newBufferedWriter since camel-2.20.0. So your
> example is failing with camel > 2.20.0
> For Red Hat Fuse, it might backport this commit to their products and your
> customer should check it with them.
>
> [1]
> https://github.com/apache/camel/commit/d8c0a53b6787778661fb4cde3c59a50b5691efac#diff-47e39aad11a143d9d2c9d60998f6fd5c118118312a8efcaed65dcd0398487426
>
> On Thu, Nov 3, 2022 at 11:09 PM Ephemeris Lappis <ephemeris.lap...@gmail.com>
> wrote:
>
> > Hello again.
> >
> > Well, as I said in my previous mail, I'm not sure that Camel is the
> > single component that's involved in our issues.
> > I've built a very little java project with a very simple route that
> > does the same operation as my blueprint.
> >
> > I've got strange results.
> >
> > When running with Camel 2.17.0 (the version that fails in our
> > production system), It works (yes, very strange) !
> > But when I try switching to 2.25.4, it fails again.
> >
> > In both cases I only use Camel and the logger over a Java 1.8 runtime.
> >
> > I attach the zipped project. Perhaps someone can test it too.
> >
> > This let me think that "something" elsewhere is involved that makes
> > similar routes work or fail, and, if I'm not mistaken, perhaps we can
> > control this "something"...
> >
> > Any idea from experts ???
> >
> > Thanks again.
> >
> > Regards.
> >
> >
> >
> >
> > Le jeu. 3 nov. 2022 à 12:30, Otavio Rodolfo Piske
> > <angusyo...@gmail.com> a écrit :
> > >
> > > Hi,
> > >
> > > I wouldn't know about the reason for this behavior, but my suggestion is
> > to
> > > try with a newer version of Camel. Camel version 2.x is not maintained
> > > anymore.
> > >
> > > Please, can you try with a newer version of Camel? Something like 3.18.3
> > > ... and let us know if this is still a problem?
> > >
> > > Kind regards
> > >
> > > On Thu, Nov 3, 2022 at 9:30 AM Ephemeris Lappis <
> > ephemeris.lap...@gmail.com>
> > > wrote:
> > >
> > > > Hello.
> > > >
> > > > The issue happens on different versions of Camel with different
> > > > configurations :
> > > > - 2.25.4 / karaf 4.3.6 / jdk 1.8
> > > > - 2.25.4 / karaf 4.4.1 / jdk 18
> > > > - 2.17.0 / fuse 6.3 / jdk 1.8
> > > >
> > > > The last one is the production environment where the issue is
> > critical...
> > > >
> > > > What's strange, is that the same little blueprint works as expected
> > > > without any error with :
> > > > - 2.16.5 / karaf 4.0.9 (smx 7.0.1) / jdk 1.8
> > > >
> > > > Any idea of the origin of the issue, and if a workaround exists : an
> > > > endpoint option, some configuration, anything ?
> > > >
> > > > Thanks.
> > > >
> > > > Regards.
> > > >
> > > > Le mer. 2 nov. 2022 à 18:51, Otavio Rodolfo Piske
> > > > <angusyo...@gmail.com> a écrit :
> > > > >
> > > > > Hello,
> > > > >
> > > > > What version of Camel are you using?
> > > > >
> > > > > Kind regards
> > > > >
> > > > > On Wed, Nov 2, 2022 at 3:22 PM Ephemeris Lappis <
> > > > ephemeris.lap...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello again.
> > > > > >
> > > > > > Sorry to answer my own question...
> > > > > >
> > > > > > I've read again the file component documentation looking for any
> > > > > > solution to fix ou issue, and I've seen that this is exactly the
> > > > > > example that it's given for using charset :
> > > > > >
> > > > > > The charset option allows for configuring an encoding of the files
> > on
> > > > > > both the consumer and producer endpoints. For example if you read
> > > > > > utf-8 files, and want to convert the files to iso-8859-1, you can
> > do:
> > > > > >
> > > > > > from("file:inbox?charset=utf-8")
> > > > > > .to("file:outbox?charset=iso-8859-1")
> > > > > >
> > > > > > And this seems to be always failing with the
> > > > > > UnmappableCharacterException when a character from the input has no
> > > > > > mapped character in the output charset...
> > > > > >
> > > > > > Nobody has reported it yet ?
> > > > > >
> > > > > > Thanks for your feedback.
> > > > > >
> > > > > > Regards.
> > > > > >
> > > > > > Le ven. 28 oct. 2022 à 16:59, Ephemeris Lappis
> > > > > > <ephemeris.lap...@gmail.com> a écrit :
> > > > > > >
> > > > > > > Hello.
> > > > > > >
> > > > > > > I've been using the File component for years, and discover today
> > a
> > > > > > > strange exception when using a charset on a producer endpoint.
> > > > > > >
> > > > > > > Example :
> > > > > > >
> > > > > > > <route id="pma-file">
> > > > > > > <from
> > > > > >
> > > >
> > uri="file:/TMP/PMA/input?delete=true&charset=UTF-8&moveFailed=.BUG"/>
> > > > > > > <log message="PMA : processing ${file:name} (length
> > > > ${file:length})..."/>
> > > > > > > <to uri="file:/TMP/PMA/output?charset=ISO-8859-1"/>
> > > > > > > </route>
> > > > > > >
> > > > > > > If I put a file that contains only characters that exist in
> > > > > > > ISO-8859-1, no problem.
> > > > > > > If the file contains an € (euro) for example, an exception is
> > thrown
> > > > > > > (see at the end of the mail) :
> > > > > > > java.nio.charset.UnmappableCharacterException: Input length = 1
> > > > > > >
> > > > > > > A trivial java program, with the same jvm on the same machine,
> > that
> > > > > > > reads a file in UTF-8 and writes its content into another file
> > with
> > > > > > > ISO-8859-1 only replaces the missing unmapped characters with ?
> > > > > > >
> > > > > > > Is it a bug (I can't imagine nobody saw that before) or just a
> > > > missing
> > > > > > > option on the endpoint ?
> > > > > > > I've not found any option for that...
> > > > > > >
> > > > > > > Thanks for your help.
> > > > > > >
> > > > > > > Regards.
> > > > > > >
> > > > > > > The exception :
> > > > > > >
> > org.apache.camel.component.file.GenericFileOperationFailedException:
> > > > > > > Cannot store file: C:\TMP\PMA\output\F1.txt
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:359)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:305)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:80)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:452)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:221)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:185)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > >
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > [?:1.8.0_242]
> > > > > > > at
> > > > > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > > > > > [?:1.8.0_242]
> > > > > > > at
> > > > > >
> > > >
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > > > > > [?:1.8.0_242]
> > > > > > > at
> > > > > >
> > > >
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > > > > > [?:1.8.0_242]
> > > > > > > at
> > > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > [?:1.8.0_242]
> > > > > > > at
> > > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > [?:1.8.0_242]
> > > > > > > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
> > > > > > > Caused by: java.nio.charset.UnmappableCharacterException: Input
> > > > length =
> > > > > > 1
> > > > > > > at
> > > > > > java.nio.charset.CoderResult.throwException(CoderResult.java:282)
> > > > > > > ~[?:1.8.0_242]
> > > > > > > at
> > sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:285)
> > > > > > > ~[?:1.8.0_242]
> > > > > > > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> > > > > > ~[?:1.8.0_242]
> > > > > > > at
> > > > java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
> > > > > > > ~[?:1.8.0_242]
> > > > > > > at
> > > > java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129)
> > > > > > > ~[?:1.8.0_242]
> > > > > > > at java.io.BufferedWriter.flush(BufferedWriter.java:253)
> > > > > > ~[?:1.8.0_242]
> > > > > > > at org.apache.camel.util.IOHelper.copy(IOHelper.java:250)
> > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.component.file.FileOperations.writeFileByReaderWithCharset(FileOperations.java:421)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > >
> > > >
> > org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:337)
> > > > > > > ~[!/:2.25.4]
> > > > > > > ... 22 more
> > > > > > > Suppressed:
> > java.nio.charset.UnmappableCharacterException:
> > > > > > > Input length = 1
> > > > > > > at
> > > > > > > java.nio.charset.CoderResult.throwException(CoderResult.java:282)
> > > > > > > ~[?:1.8.0_242]
> > > > > > > at
> > > > > > > sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:285)
> > > > > > > ~[?:1.8.0_242]
> > > > > > > at
> > > > > > > sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> > ~[?:1.8.0_242]
> > > > > > > at
> > > > > > > java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
> > > > > > > ~[?:1.8.0_242]
> > > > > > > at
> > > > > > > java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129)
> > > > > > > ~[?:1.8.0_242]
> > > > > > > at
> > > > > > > java.io.BufferedWriter.close(BufferedWriter.java:265)
> > ~[?:1.8.0_242]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.component.file.FileOperations.writeFileByReaderWithCharset(FileOperations.java:422)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:337)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:305)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:80)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > >
> > org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
> > > > > > > ~[!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > > > org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > > > org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:452)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:221)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:185)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
> > > > > > > [!/:2.25.4]
> > > > > > > at
> > > > > > >
> > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > [?:1.8.0_242]
> > > > > > > at
> > > > > > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > > > > > [?:1.8.0_242]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > > > > > [?:1.8.0_242]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > > > > > [?:1.8.0_242]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > [?:1.8.0_242]
> > > > > > > at
> > > > > > >
> > > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > [?:1.8.0_242]
> > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > [?:1.8.0_242]
> > > > > > > 1
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Otavio R. Piske
> > > > > http://orpiske.net
> > > >
> > >
> > >
> > > --
> > > Otavio R. Piske
> > > http://orpiske.net
> >
