[ https://issues.apache.org/jira/browse/CAMEL-9274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004852#comment-15004852 ]
Aaron Whiteside edited comment on CAMEL-9274 at 11/13/15 10:53 PM: ------------------------------------------------------------------- I've noticed a lot of inconsistency, some DataFormat's just return a InputStream others that could return an InputStream read it into a buffer then return that buffer, and then we have StreamCaching, which is intended to solve the issue of buffering too much data in memory/heap. But a lot of those places we use StreamCaching we could also just return the InputStream in question.. I've been thinking of adding support to the DataFormat's I care about to configure the desired behavior, return byte[], raw InputStream or stream cached InputStream. And I must admit I was tempted to change the behavior of existing DataFormat's to return their raw InputStream's instead of fixing StreamCaching, but I figured that might break someone else. What are your thoughts? My use case is, decrypt, unzip, and split file content using a streaming iterator. So having a bunch of wrapped InputStreams would be much more efficient than buffering things in a StreamCache twice (decrypt, unzip). was (Author: aaronjwhiteside): I've noticed a lot of inconsistency, some DataFormat's just return a InputStream others that could return an InputStream read it into a buffer then return that buffer, and then we have StreamCaching, which is intended to solve the issue of buffering too much data in memory/heap. But a lot of those places we use StreamCaching we could also just return the InputStream in question.. I've been thinking of adding support to the DataFormat's I care about to configure the desired behavior, return byte[], raw InputStream or stream cached InputStream. And I must admit I was tempted to change the behavior of existing DataFormat's to return their raw InputStream's instead of fixing StreamCaching, but I figured that might break someone else. What are your thoughts? > StreamCache does not work correctly with MarshalProcessor and pretty much all > DataFormats > ----------------------------------------------------------------------------------------- > > Key: CAMEL-9274 > URL: https://issues.apache.org/jira/browse/CAMEL-9274 > Project: Camel > Issue Type: Improvement > Components: camel-core > Affects Versions: 2.16.0 > Reporter: Aaron Whiteside > Assignee: Claus Ibsen > Fix For: 2.17.0 > > Attachments: CAMEL_9274.patch, CAMEL_9274_v2.patch, > CAMEL_9274_v3.patch > > > StreamCache does not work correctly with MarshalProcessor and pretty much all > DataFormats > # Documentation states that stream caching can be turned on, on a per route > basis. The places were we actually try and utilize stream caching only checks > if the global option is turned on (MarshalProcessor + PGPKeyAccessDataFormat, > I can't find any other meaningful attempts to use stream caching anywhere > else) not the per route setting. > # Secondly, if we ignore the first problem, only one data formatter tries to > use stream caching for unmarshaling: PGPKeyAccessDataFormat. If you try and > unzip a large zip file it's contents will be decompressed into memory. The > same goes for any other large data formatter conversion (unmarshaling, only). > Attached is a patch that corrects the first problem and part of the second > problem (ZipFileDataFormat, ZipDataFormat and PGPKeyAccessDataFormat). > A separate issue should probably be opened to fix all the remaining > DataFormat implementation to support Stream Caching. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)