The real problem is: if split expression cannot do streaming operation, the memory cost cannot be avoid; Here the xpath is example which cannot do streaming. Csv will be another example. It means we cannot process a very large csv file (is it?I am not sure, but it seems so) So how about supporting "batch" operations on the camel's component? I am not sure if this is suitable for camel. But some other commercial adapter's does support such a function;
Also the batch options is not only apply for file component, but also apply to other components, etc: a SQL component which should output the result batched; ----original ----- Sender: Charles Moulliard [mailto:cmoulli...@gmail.com] Date: 2010/12/14 19:22 Receiver: users@camel.apache.org Subject: Re: Splitting big XML files using xpath() and streaming() Hi Christian, Interesting discussion that you have started where we reach the border / limit about what camel should do in a messaging approach instead of a batch process. Regarding to your question about using a stylesheet to split the XML files into multiple small files, I'm not quite sure that this is the right solution as we have to check what happens between all the files individually before to answer to the question 'Is my file valid' ? We will probably speed up the process and reduce memory consumption but increase complexicity of the solution. If nevertheless, there is no other alternatives, we could use a SEDA processor to split the big file in small files, process them individually and use an aggregator to check if each file has been validated/processed correctly Regards, Charles On 14/12/10 10:54, Christian Müller wrote: > @Claus: As what I found, using XPath in a streaming mode is not possible out > of the box in Java 5 or 6 (XPath 1.0 needs the DOM XML Document). JAXP 1.4 > (part of Java 6) includes the StAX API and can be used in Java 5. But than > we have to parse the XPath expression by our self and use the Iterator API > (XMLEventReader)... I think this is not what we want... I will update the > wiki page with the information, that the streaming() mode can not be used in > conjunction with xpath(). > > @Charles: Thanks for the link. Do you plan to use a style sheet and split a > big xml file into multilpe small files? Or is it possible with the pure > Saxon Java API (without a style sheet) which returns an iterator? This could > be also the solution for our requirements (pure Java API which we can use in > a custom processor). > > Cheers, > Christian >
