-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Just a shot in the dark here, but in the Camel Splitter doc:
https://camel.apache.org/splitter.html parallelProcessing is marked as default off. In other words, could it be that you have each record being processed sequentially and are therefore not taking any advantage of multithreading? Perhaps you could try <camel:split streaming="true" parallelProcessing="true"> and see if there is any improvement. If there is, you could then take advantage of the executorServiceRef property to set up a custom threadpool to really tune your use of processing resources. - --- A. Soroka Software & Systems Engineering Online Library Environment the University of Virginia Library On Feb 15, 2012, at 8:29 AM, vs.souza wrote: > Hello fellows, > > I created a bundle to load a CSV file, validate some information and send it > to an ActiveMQ queue. The problem is that my CSV file has 30.000.000 > registers and about 1.2 GB and my bundle is taking about 12 hours to load > 750.000 itens to the queue. My ServiceMix is 4.4.3 and I am loading the file > woking wit camel and bindy. Bellow I post my camel-context.xml and my bindy > class: > > camel-context.xml: > > <?xml version="1.0" encoding="UTF-8"?> > > <beans xmlns="http://www.springframework.org/schema/beans"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xmlns:camel="http://camel.apache.org/schema/spring"; > xsi:schemaLocation=" > http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans.xsd > http://camel.apache.org/schema/spring > http://camel.apache.org/schema/spring/camel-spring.xsd";> > > <bean id="bindyDataFormat" > class="org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat"> > <constructor-arg value="com.test.integration.camel.spring.poc.file"/> > </bean> > > <camel:camelContext xmlns="http://camel.apache.org/schema/spring";> > <camel:package>com.test.integration.camel.spring.poc</camel:package> > <camel:route> > <camel:from > uri="file:/home/jedimaster/Java-Env/Sandbox/From?delete=true"/> > <camel:log message="Started unmarshalling file ${file:name} at > ${date:now:hh:MM:ss.SSS}..."/> > <camel:split streaming="true"> > <camel:tokenize token="\n"/> > <camel:unmarshal ref="bindyDataFormat"/> > <camel:to uri="activemq:filemove-events"/> > </camel:split> > <camel:log message="Finished unmarshalling file ${file:name} at > ${date:now:hh:MM:ss.SSS}..."/> > </camel:route> > > > </camel:camelContext> > > </beans> > > bindy bean: > > package com.test.integration.camel.spring.poc.file; > > import java.io.Serializable; > > import org.apache.camel.dataformat.bindy.annotation.CsvRecord; > import org.apache.camel.dataformat.bindy.annotation.DataField; > > @CsvRecord(separator=",", quote="\"") > public class CSVEventRecordBean implements Serializable{ > > private static final long serialVersionUID = -8806841912643394977L; > > @DataField(pos=1) > private String eventDate; > > @DataField(pos=2) > private String userId; > > @DataField(pos=3) > private String systemId; > > public String getEventDate() { > return eventDate; > } > > public void setEventDate(String eventDate) { > this.eventDate = eventDate; > } > > public String getUserId() { > return userId; > } > > public void setUserId(String userId) { > this.userId = userId; > } > > public String getSystemId() { > return "Bean Generated: " + systemId; > } > > public void setSystemId(String systemId) { > this.systemId = systemId; > } > > } > > > How can I improve the performance considerably to make it faster? Do you > have any suggestions? > > Cheers. > > -- > View this message in context: > http://servicemix.396122.n5.nabble.com/Performance-issues-to-load-csv-file-tp5486044p5486044.html > Sent from the ServiceMix - User mailing list archive at Nabble.com. -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJPO7vAAAoJEATpPYSyaoIkr+oH/1jdTe3ueq38+C3/wy0GCX2E QZRDl7YQYuDpS7eQADTMRxb0tJyqTOaLXnmLaiVpCyymdJOv2hvlZa0tFegdXqMv qnE4czUTe5zJi/KXxnrsVuS1yIJyRK7E8IXurkTPecb8c1eRiHMCE1247jQ6LPcG 0/3e5SLGQzz4/0p4wob1f4eJGn7+u4OIFYirzUMMaUX7XZ3IwnmVrntCU6oTfAkB Xqa8PevjGB/l34GCHHfV2LhAhRgIyIx1vK8SJja748Flkn18NeLGIkUXfsz9gyXM fRtVxLVMPXklbdUvQUhtfoqgEI7cJ7v/PfAleFlEySeC7bvphYveWbdpcMuy53A= =yZLh -----END PGP SIGNATURE-----
