Hi,
We have some large files which we have to process from an FTP server.
Currently we have to run camel with -Xmx2048m to allow these files to
process without camel failing with an (unreported) OutOfMemory error
(which we picked up with -XX:+HeapDumpOnOutOfMemoryError)
Reading the documentation points in the direction of not using the
default idempotent consumer[1]. I suppose I will have to use some
kind of JPA or filebased idempotent store - I don't like the idea of
this, so I'm trying to come up with a way of avoiding it.
We currently have the following Spring route:
<camel:route>
<camel:from ref="stock-schedule"/>
</camel:route>
<camel:route id="stock-route-download" startupOrder="0">
<camel:from ref="stock-ftp"/>
<camel:to ref="stock-file"/>
</camel:route>
<camel:route id="stock-file-processing-route">
<camel:from ref="stock-file-consumer"/>
<camel:bean ref="stockUnmarshaller" method="unmarshall"/>
<camel:split streaming="true">
<camel:method bean="stockTransformer" method="transform"/>
<camel:bean ref="persister" method="persist"/>
</camel:split>
</camel:route>
In this we are consuming a 100+Mb file in one gulp, then splitting and
streaming the generated objects to the db. If I was to move the
splitter upstream:
<camel:route id="stock-file-processing-route">
<camel:from ref="stock-file-consumer"/>
<camel:split streaming="true">
<camel:bean ref="stockUnmarshaller" method="unmarshall"/>
<camel:method bean="stockTransformer" method="transform"/>
<camel:bean ref="persister" method="persist"/>
</camel:split>
</camel:route>
Would this reduce the memory consumption? This is just one of many
routes, and I'm concerned about the apparent wastefulness of camel at
the moment (although I'm fairly convinced that it's my config that is
incorrect).
[1]http://camel.apache.org/idempotent-consumer.html
