Re: Best Strategy to process a large number of rows in File

[Michele]

Hi Brad,

first of all thank you very much for the time you dedicate me.

Are you getting the entire file in memory? I think so. I thought BeanIO
worked in lazy mode...

A question, I noticed that in some file Splitter doesn't work with
<from uri="file:inbox"/> 
  <split streaming="true"> 
    <tokenize token="\n" /> 
    <to uri="activemq:queue:store"/> 
  </split> 

but It required a convertBodyTo type="java.lang.String" before split, but we
don't want this (entire file in memory). How can I split file without
converter?

I tried to introduce some changes at Route as you suggested with a good use
of memory avoiding OOM.

Current Route:

<route id="FileRetriever_Route">
<from
uri="{{uri.inbound}}?scheduler=quartz2&amp;scheduler.cron={{poll.consumer.scheduler}}&amp;scheduler.triggerId=FileRetriever&amp;scheduler.triggerGroup=IF_SAP{{uri.inbound.options}}"
/>
<split streaming="true" executorServiceRef="ThreadPoolExecutor">
        <tokenize token="\n" />                                 
        <choice>
                <when>
                        <simple></simple>
                        <log message="Store Message in Queue 
${in.header.CamelSplitIndex} -
${body}" loggingLevel="DEBUG" />
                        <to uri="activemq:queue:SAP.Product" />
                </when>
                <otherwise>
                        <log message="Invalid Row ${in.header.CamelSplitIndex} 
- Discarded:
${body}" />
                </otherwise>
        </choice>
</split>
</route>
<route id="ProcessMessageData_Route" errorHandlerRef="DLQErrorHandler" >
                        <from
uri="activemq:queue:SAP.Product?destination.consumer.prefetchSize=1" />
        
                        <throttle timePeriodMillis="1000" asyncDelayed="true">
                                <constant>3</constant>
                                
                                <log message="START - ProcessMessageData_Route" 
/>
                                
                                
                                <unmarshal ref="RecordParser" />
                                <setBody><simple>${body[0]}</simple></setBody>
                                <to
uri="dozer:transform?mappingFile=file:{{crt2.apps.home}}{{dozer.mapping.path}}&amp;targetModel=java.util.LinkedHashMap"
/>
                                <marshal ref="Gson" />  
                                
                                <enrich uri="direct:crm-login" 
strategyRef="OAuthStrategy" />                           
                                
                                <recipientList>
                                        <header>CompleteActionPath</header>
                                </recipientList>

                        </throttle>
</route>

Thanks a lot again

Best Regards

Michele




--
View this message in context: 
http://camel.465427.n5.nabble.com/Best-Strategy-to-process-a-large-number-of-rows-in-File-tp5779856p5781251.html
Sent from the Camel - Users mailing list archive at Nabble.com.