could you elaborate a little bit on the xml mapper? how do you get from data lines to xml? E.g. instead of generating strings for concatenation you could create StAX or SAX events and stream to your output. Why do you need to append the xml as strings? could you append to a file, write to a stream …. Are all the input lines transformed into xml and then appended to the same result ?
To me there are a number of different isolated issues which might not be related to Apache Camel itself but to the steps within your process and how you handle the data. Maybe you break the resolvment into parts. 1st ensure that you read the data as efficient as possible i.e. verify if there is really an IO issue. 2nd look at how to best get from data lines to xml. Maruan Sahyoun Am 22.02.2013 um 08:26 schrieb cristisor <cristisor...@yahoo.com>: > I get the OOME when because I send each line to the xml mapper and I append > the output of the mapper to a StringBuilder. From a short line I get a big > XML and StringBuilder throws OOME if I append too many outputs. Because I > can't risk of having the OOME if the xml will get bigger in time I should be > able to read 500 lines at a time and send each of time independently or send > 500 lines at once but map each of them to an xml and send the xml alone to > the second stage xml mapper. Do you get my point now? > > Thanks. > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Large-file-processing-with-Apache-Camel-tp5727977p5727997.html > Sent from the Camel - Users mailing list archive at Nabble.com.
