Hi You should split the file before you do the CSV thingy. The splitter can work streaming on the file out of the box. So just split it by new line, or lets say per 10.000 lines etc.
Then you avoid sucking in all the file, or having all the CSV records in one big Exchange. On Thu, Aug 19, 2010 at 8:23 AM, vcheruvu <v_cher...@hotmail.com> wrote: > > I have changed my logging level to INFO but it didn't solve memory issue. I > have turned on JConsole noticed that, memory shoots up to 1.05GB when Camel > reads in 218K lines, 45 MB file and then throws heap memory issue with > Jconsole RMI which is trying to get memory info of the application. > > I am bit puzzled as to what happening in the Camel-File component when it > reads in all the line from the csv file. > > I could only get around the issue only if I split the file into 10K lines > per file. In this case, camel reads all the 20 files one by one, then it > reaches memory usage upto 900MB and but doesn't crash with heap memory > issue. Why didn't heap memory occurred in this scenario? is this because, > JVM had sufficient time to GC objects? or application JVM settings for GC is > poor? > > -Xms1024m -Xmx1024m -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=8 > -XX:NewSize=128m -XX:MaxNewSize=128m -XX:+UseParNewGC > -XX:+CMSParallelRemarkEnabled -XX:PermSize=64m -XX:MaxPermSize=64m > -XX:+UseAdaptiveSizePolicy > > I have attached jconsole memory graph for both cases. Both cases , I have > set the same JVM settings. > > I guess it will be nice addition if we could have option for file consumer > to limit number of lines to be read in for every poll regardless of number > files. This will give some control on application to manage memory, > throughput and stability. > http://camel.465427.n5.nabble.com/file/n2640474/heapspace-singlefile.bmp > heapspace-singlefile.bmp > http://camel.465427.n5.nabble.com/file/n2640474/heapspace-multifile.bmp > heapspace-multifile.bmp > -- > View this message in context: > http://camel.465427.n5.nabble.com/Java-heap-space-issue-with-reading-large-CSV-file-tp2638903p2640474.html > Sent from the Camel - Users mailing list archive at Nabble.com. > -- Claus Ibsen Apache Camel Committer Author of Camel in Action: http://www.manning.com/ibsen/ Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/ Twitter: http://twitter.com/davsclaus
