Thank you everybody for your help. After reading and trying, I found how to implement splitter and aggregators and I managed to achieve my scope.
Here is the new status: 1. read 500 lines from the file and send them in an exchange to the next service unit, to a certain endpoint according to the file type 2. this service unit first splits the exchange in 500 exchanges and then sends each exchange to the same processor that does the step1 xml mapping 3. after the processor I aggregate the exchanges back by concatenating the out xmls, but only 50 at a time so that I won't get a OOME or something else 4. after the aggregator concatenates 50 exchanges or a certain amount of time passes I send the new exchange to the next service unit, step2 xml mapping 5. now I split the exchange again in 50 different exchanges and each of them goes to the processor, where the final xml is being created 6. the final xml is sent in the exchange to the database service unit, where I first aggregate 300 exchanges by grouping them and then I send the grouped exchange to the final processor so that I can get the data from all 300 exchanges and do a batch insert/update with a prepared statement and 300 rows Now I have 3 more questions, maybe you can give me some tips: 1. the server has 4 cores, can I do parallel processing so that I can have 4 lines processed in parallel, 4 xmls generated in parallel, etc? 2. I think that I can compress the xml strings, so that I might be able to send hundreds of xmls instead of 50 or 100, am I right, should I investigate this path? 3. in the final service unit, the one that inserts into the database, we use a ChoiceProcessor to select the specific processor, according to the DAO type, so that I can make the insert/update procedure. This choice processor has a list of FilterProcessors and each FilterProcessor has the actual DAO processor. How can I insert an aggregator between the FilterProcessor and the final DAO processor without using: route definition -> choose -> when() -> aggregate -> process -> otherwise -> aggregate -> process Sorry for so many questions but I woke up one day with a very slow application and I have to make it work faster as soon as possible. Thanks again. -- View this message in context: http://camel.465427.n5.nabble.com/Large-file-processing-with-Apache-Camel-tp5727977p5728045.html Sent from the Camel - Users mailing list archive at Nabble.com.
