Hi Okay I have added some tests to Camel to see the performance for such use-case. https://svn.apache.org/repos/asf/camel/trunk/camel-core/src/test/java/org/apache/camel/component/file/stress/
This test uses the Camel file producer https://svn.apache.org/repos/asf/camel/trunk/camel-core/src/test/java/org/apache/camel/component/file/stress/FileProducerAppendManyMessagesTest.java And this uses java code by reusing the same output stream for the writes https://svn.apache.org/repos/asf/camel/trunk/camel-core/src/test/java/org/apache/camel/component/file/stress/FileProducerAppendManyMessagesFastTest.java The latter is much faster, as it uses the same stream for all the writes. The former is slow because a new stream is obtained, and positioned to the end of the file, and so forth. On Fri, Nov 9, 2012 at 4:01 PM, Gonzalo Vasquez <gvasq...@altiuz.cl> wrote: > I'm running a route that basically adds a character per line to a plain text > file, but it's taking to long, and it seems that it's due to some kind of > buffering issue when reading/writing from disk. > > I'm processing a 5MB file (attached as DC_FACCL132_0000 > MORA_1075_16-10-2012_19-09-47_15.txt.zip), with the corresponding XSL > template (also attached). > > It's taking for ever to process such a file, I understand I'm tokenizing on > line breaks, which could be the source of the problem as there are many > lines in the file (48198 exactly), but when running jvisualvm (see attached > images/snapshot)I can see the writing op is invoked 20386 times, which seem > not related to the line count. Is there an output buffer size that I can > configure? Or something like that? > > This is the route: > <camel:route id="pager" autoStartup="true"> > <camel:from > uri="file:///tmp/in?charset=Windows-1252&move=${file:parent}/../paged/${file:name.noext}.paged.ack&preMove=${file:name.noext}-${date:now:yyyyMMddHHmmssSSS}.${file:ext}" > /> > <camel:split streaming="true" parallelProcessing="false"> > <camel:tokenize token="\n" /> > <camel:to uri="bean:pager" /> > <camel:to > uri="file:///tmp/paged?charset=utf8&fileName=${file:name.noext}.paged&fileExist=Append" > /> > </camel:split> > </camel:route> > > This is the referenced bean: > > <bean id="pager" class="cl.altiuz.reports.etl.TextProcessor"> > <property name="xsltPath" > value="/Users/gonzalovasquez/Documents/workspace/altiuz-reports/reports-etl/xsl/pager.xsl" > /> > <property name="param" value="C.*PAG.* 1" /> > </bean> > > Camel versión is 2,10.1, and happens both on OSX & MS Windows, so I think > isn't a platform dependent problem, but a configuration one. > > Any ideas? Any thing else that I should send? > > Thanks! > > Gonzalo Vásquez Sáez > Gerente Investigación y Desarrollo (R&D) > Altiuz Soluciones Tecnológicas de Negocios Ltda. > Av. Nueva Tajamar 555 Of. 802, Las Condes > (56-2) 335 2461 > gvasq...@altiuz.cl > http://www.altiuz.cl > > > > > > > > -- Claus Ibsen ----------------- Red Hat, Inc. FuseSource is now part of Red Hat Email: cib...@redhat.com Web: http://fusesource.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen
