[
https://issues.apache.org/jira/browse/CAMEL-20938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868939#comment-17868939
]
Ryan Ikhlef commented on CAMEL-20938:
-------------------------------------
Hello,
I tested the fix in Camel version 4.7.0, and I believe it doesn’t fully address
the issue. While the error is no longer thrown, we still don’t receive a proper
stream object when using the “disableStreamCache” option. Instead, the entire
response is loaded into the Java heap, which is problematic for us when dealing
with large API responses (exceeding 1 GB in size).
Upon reviewing the code you committed for the fix, it appears that when Camel
is in streaming mode, it consumes the stream and copies it into a byte array.
I’ve included some screenshots from Visual VM for the same routes you provided
to reproduce the issue. The only difference is that I’m not requesting a
website like Camel docs; instead, I’m using an API with substantial response
data (which I can’t share, but I can generate a similar-sized JSON if needed).
In the first screenshot, you can see the byte array built by Camel. As you’ll
notice, the current size matches my data size:
!image-2024-07-26-14-34-59-635.png!
In this second screenshot, I share the data inside the body’s exchange just
before the .log("${body}"). We found our byte array:
!image-2024-07-26-14-36-27-046.png!
Finally, I want to show you the heap of my small app:
!image-2024-07-26-14-36-58-094.png!
Maybe I’m doing something wrong, but as it’s written in the Camel HTTP
component documentation: "If setting this option to true, then the producers
will not cache the response body stream but use the response stream as-is as
the message body." I expected to get the stream in my body to consume it myself.
Best regards,
> camel-http - Using disableStreamCache=true cannot read body later due to
> stream closed
> --------------------------------------------------------------------------------------
>
> Key: CAMEL-20938
> URL: https://issues.apache.org/jira/browse/CAMEL-20938
> Project: Camel
> Issue Type: Bug
> Components: camel-http
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Minor
> Fix For: 4.4.4, 4.7.0
>
> Attachments: image-2024-07-26-14-34-59-635.png,
> image-2024-07-26-14-36-27-046.png, image-2024-07-26-14-36-58-094.png
>
>
> Reported in mailing list.
> {code}
> 2024-07-03 08:47:26.061 WARN 30864 --- [ - timer://test]
> ache.camel.component.timer.TimerConsumer : Error processing exchange.
> Exchange[8164CC526FF834E-0000000000000000]. Caused by:
> [org.apache.camel.TypeConversionException - Error during type conversion from
> type: org.apache.hc.client5.http.entity.LazyDecompressingInputStream to the
> required type: java.lang.String with value
> org.apache.hc.client5.http.entity.LazyDecompressingInputStream@6f9df776 due
> to java.io.IOException: Attempted read on closed stream.]
> org.apache.camel.TypeConversionException: Error during type conversion from
> type: org.apache.hc.client5.http.entity.LazyDecompressingInputStream to the
> required type: java.lang.String with value
> org.apache.hc.client5.http.entity.LazyDecompressingInputStream@6f9df776 due
> to java.io.IOException: Attempted read on closed stream.
> at
> org.apache.camel.converter.CamelBaseBulkConverterLoader.convertTo(CamelBaseBulkConverterLoader.java:65)
> ~[camel-base-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.spi.BulkTypeConverters.convertTo(BulkTypeConverters.java:122)
> ~[camel-api-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.tryCachedConverters(CoreTypeConverterRegistry.java:416)
> ~[camel-base-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(CoreTypeConverterRegistry.java:375)
> ~[camel-base-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertToAndStat(CoreTypeConverterRegistry.java:269)
> ~[camel-base-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.convertTo(CoreTypeConverterRegistry.java:164)
> ~[camel-base-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.support.ExpressionAdapter.evaluate(ExpressionAdapter.java:52)
> ~[camel-support-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.support.builder.ExpressionBuilder$62.evaluate(ExpressionBuilder.java:2105)
> ~[camel-support-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.support.ExpressionAdapter.evaluate(ExpressionAdapter.java:45)
> ~[camel-support-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.processor.LogProcessor.process(LogProcessor.java:71)
> ~[camel-core-processor-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.handleFirst(RedeliveryErrorHandler.java:440)
> ~[camel-core-processor-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:416)
> ~[camel-core-processor-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.doRun(DefaultReactiveExecutor.java:199)
> [camel-base-engine-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeReactiveWork(DefaultReactiveExecutor.java:189)
> [camel-base-engine-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.tryExecuteReactiveWork(DefaultReactiveExecutor.java:166)
> [camel-base-engine-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
> [camel-base-engine-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59)
> [camel-base-engine-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:163)
> [camel-core-processor-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.impl.engine.CamelInternalProcessor.processNonTransacted(CamelInternalProcessor.java:346)
> [camel-base-engine-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:322)
> [camel-base-engine-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:293)
> [camel-timer-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.component.timer.TimerConsumer$1.doRun(TimerConsumer.java:164)
> [camel-timer-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:136)
> [camel-timer-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at java.base/java.util.TimerThread.mainLoop(Timer.java:566) [?:?]
> at java.base/java.util.TimerThread.run(Timer.java:516) [?:?]
> Caused by: java.io.IOException: Attempted read on closed stream.
> at
> org.apache.hc.core5.http.io.EofSensorInputStream.isReadAllowed(EofSensorInputStream.java:107)
> ~[httpcore5-5.2.4.jar:5.2.4]
> at
> org.apache.hc.core5.http.io.EofSensorInputStream.read(EofSensorInputStream.java:116)
> ~[httpcore5-5.2.4.jar:5.2.4]
> at
> java.base/java.util.zip.CheckedInputStream.read(CheckedInputStream.java:59)
> ~[?:?]
> at
> java.base/java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:266)
> ~[?:?]
> at
> java.base/java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:258)
> ~[?:?]
> at
> java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:164)
> ~[?:?]
> at
> java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79) ~[?:?]
> at
> java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:91) ~[?:?]
> at
> org.apache.hc.client5.http.entity.GZIPInputStreamFactory.create(GZIPInputStreamFactory.java:61)
> ~[httpclient5-5.2.1.jar:5.2.1]
> at
> org.apache.hc.client5.http.entity.LazyDecompressingInputStream.initWrapper(LazyDecompressingInputStream.java:51)
> ~[httpclient5-5.2.1.jar:5.2.1]
> at
> org.apache.hc.client5.http.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:69)
> ~[httpclient5-5.2.1.jar:5.2.1]
> at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:270)
> ~[?:?]
> at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:313)
> ~[?:?]
> at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:188)
> ~[?:?]
> at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
> ~[?:?]
> at java.base/java.io.BufferedReader.fill(BufferedReader.java:162) ~[?:?]
> at java.base/java.io.BufferedReader.read1(BufferedReader.java:213)
> ~[?:?]
> at java.base/java.io.BufferedReader.read(BufferedReader.java:287) ~[?:?]
> at java.base/java.io.Reader.read(Reader.java:250) ~[?:?]
> at org.apache.camel.util.IOHelper.toString(IOHelper.java:144)
> ~[camel-util-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at org.apache.camel.util.IOHelper.toString(IOHelper.java:128)
> ~[camel-util-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at org.apache.camel.util.IOHelper.toString(IOHelper.java:124)
> ~[camel-util-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.converter.IOConverter.toString(IOConverter.java:186)
> ~[camel-base-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.converter.IOConverter.toString(IOConverter.java:201)
> ~[camel-base-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.converter.CamelBaseBulkConverterLoader.doConvertTo(CamelBaseBulkConverterLoader.java:345)
> ~[camel-base-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> at
> org.apache.camel.converter.CamelBaseBulkConverterLoader.convertTo(CamelBaseBulkConverterLoader.java:56)
> ~[camel-base-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]
> ... 24 more
> {code}
> When running
> {code}
> import org.apache.camel.Exchange;
> public class HttpBug extends org.apache.camel.builder.RouteBuilder {
> @Override
> public void configure() throws Exception {
> getContext().getTypeConverterRegistry();
> //@formatter:off
> from("timer:test?repeatCount=1")
> .setHeader(Exchange.HTTP_METHOD, constant("GET"))
> .setHeader(Exchange.HTTP_URI,
> constant("https://camel.apache.org/components/4.4.x/http-component.html";))
> .to("http:myHttpRequest?disableStreamCache=true")
> .log("body : ${body}")
> .end();
> //@formatter:on
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
