#1 is a fine option as you need to use the splitter in streaming mode
to split 1 message into N messages.
In #2 you just convert the body from one type to another, its still 1
message from Camel point of view. And only 1 message send to the AMQ.
So go with #1.
For #1 you can possible look at the tokenizer as it can split based on
tokens / line feeds etc. Though using a custom bean that returns an
Iterator is very good, in fact also what the tokenizer does as well.
On Fri, Nov 8, 2013 at 8:34 PM, brenuart <bertrand.renu...@itma.lu> wrote:
> Hello,
>
> I need to poll and read text files from a given directory. Each file holds a
> large but variable number of "Item" instances (+100K) possibly encoded on a
> multiple lines (exact number not known in advance).
>
> (1)
> My first approach was to use a custom Splitter that would take the File in
> input and return an Iterator<Item> to enable streaming of the result (and
> therefore avoid having the entire unmarshalled file content into memory).
> The route looks as follows:
>
> from("file:incoming")
> .split(new MyFileItemReaderFactory(), "read")
> .streaming()
> .to("activemq:items")
>
> Although it (seems to) work(s), I don't feel comfortable with this approach
> because "MyFileItemReaderFactory" actually serves two distinct purposes:
> - unmarshall Item instances from their text representation
> - split the original file into individual items
>
> What do you think ?
>
>
> (2)
> I was wondering if a better approach would be something like:
>
> from("file:incoming")
> .convertBodyTo(Iterator<Item)
> .to("activemq:items")
>
> Would it be better from a "Camel point of view" ?
> How is streaming handled in this case ?
> Is there any better way of handling this case ?
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Read-and-process-very-large-files-tp5742889.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cib...@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
