On Thu, Nov 24, 2011 at 3:47 PM, Billy <billy.sjob...@gmail.com> wrote: > Hi all! > > I have a requirement to transform a fairly large set of xml data that I > fetch over HTTP and then FTP it out. > After getting my first OOME I started reading up a bit on how you could > utilize the splitter with streaming and tokenizing pairs (2.9RC) which > worked well for me for splitting and transforming the individual pieces. > (Great work btw on this!) > > However, I have not found a good way to aggregate the data back again after > the transformation. My custom aggregationstrategy that I set on the splitter > simply concatinates the payloads into either a String or byte[], but this > also gives me memory problems. > > Now, the streamCaching on the route coupled with a streaming splitter solves > the splitting part, but can anyone hint me in the right direction of how I > should effectively piece the parts back during the aggregation phase? > Basically offloading the storage of the aggregated data from memory.. > > I have played around with the concept of simply appending all > split/transformed elements to a file, and ftp that out after it has been > marked complete. This works, but then I loose all my state inside the route > which I need for later reporting. >
Interesting. We got a persistent aggregator, but it will load the content back into memory when loading it back. In the long run we should possible look into adding support for a InputStream as the source from the persistent aggregator, so the body would be loaded-on-demand. In the mean time. I suggest to take a look at the claim check EIP. Which frankly is possible also a great pattern to solve these kind of problems. http://camel.apache.org/claim-check.html Where you can off load the "big payload" to a file etc. And then just keep a tiny details in memory, which has a reference to the file. And then you are done processing the aggregated message, you can delete the file. > -- > View this message in context: > http://camel.465427.n5.nabble.com/Need-tips-for-splitter-aggregator-with-large-data-tp5020356p5020356.html > Sent from the Camel - Users mailing list archive at Nabble.com. > -- Claus Ibsen ----------------- FuseSource Email: cib...@fusesource.com Web: http://fusesource.com Twitter: davsclaus, fusenews Blog: http://davsclaus.blogspot.com/ Author of Camel in Action: http://www.manning.com/ibsen/
