Hello,
I need to poll and read text files from a given directory. Each file holds a
large but variable number of "Item" instances (+100K) possibly encoded on a
multiple lines (exact number not known in advance).
(1)
My first approach was to use a custom Splitter that would take the File in
input and return an Iterator<Item> to enable streaming of the result (and
therefore avoid having the entire unmarshalled file content into memory).
The route looks as follows:
from("file:incoming")
.split(new MyFileItemReaderFactory(), "read")
.streaming()
.to("activemq:items")
Although it (seems to) work(s), I don't feel comfortable with this approach
because "MyFileItemReaderFactory" actually serves two distinct purposes:
- unmarshall Item instances from their text representation
- split the original file into individual items
What do you think ?
(2)
I was wondering if a better approach would be something like:
from("file:incoming")
.convertBodyTo(Iterator<Item)
.to("activemq:items")
Would it be better from a "Camel point of view" ?
How is streaming handled in this case ?
Is there any better way of handling this case ?
--
View this message in context:
http://camel.465427.n5.nabble.com/Read-and-process-very-large-files-tp5742889.html
Sent from the Camel - Users mailing list archive at Nabble.com.
