Hi
I read XML files in sizes up to about 400 MB with the following route - it uses
file AND streaming component.
Since it is XML the splitting is done based on a specific XML element. The
chunks are then sent to a JMS queue. Before sending it to the queue you can of
course do transformations etc to create the needed format.
from(fileEndpointUri)
.routeId(localRouteId)
.split().tokenizeXML(<XML-Elementname used for splitting>).streaming()
.to(queueEndpointUri);
Regards
Stephan
-----Ursprüngliche Nachricht-----
Von: Rajith Muditha Attapattu [mailto:rajit...@gmail.com]
Gesendet: Mittwoch, 17. Mai 2017 17:29
An: users@camel.apache.org
Betreff: Processing very large files with Camel
I have a very large flat file, say 50-100 GBs (ex daily transactions).
I'm looking at the possibility of using camel to process the flat file and
update a database Camel file and stream components come into place.
My gut reaction is to have a route that simply read 500MB to 1GB worth of data
and write to a file in a folder. (Break down the into manageable
chunks)
Another route looks at the folder and picks a file and hands it over to a
thread pool backed route to process them.
Has anybody attempted this kind of scenario?
If so have you run into challenges ?
Are there any limitations with streaming or file component? and which component
is better placed for this task?
Regards,
Rajith Muditha Attapattu <http://rajith.2rlabs.com/>
