RE: Best Strategy to process a large number of rows in File

[Michele]

Hi,

In this case file size is 20.424KB and it contains about 40000 lines.

I used the convertTo to avoid Rollback on rename file
(GenericFileOperationFailedException - only Windows).

The File is a cvs and so I defined CSV Data format like this (Transforms
each line in a map key-value)

<csv id="ReaderDataFormat" delimiter="{{in.file.delimiter}}"
skipHeaderRecord="true" ignoreEmptyLines="true" useMaps="true">
                                <header>Key</header>
                                <header>Date</header>
                                <header>Time</header>
                                <header>User1</header>
                                <header>User2</header>
                                <header>User3</header>
                                <header>Status</header>
                                <header>CertificateNumber</header>
                                <header>CA_KEY_INDEX</header>
                                <header>INJ_CARD_SN</header>
                                <header>TERM_PN</header>
                                <header>TERM_SN</header>
                                <header>Certificate</header>
                        </csv>

I used CSV DF to handle CSV payloads.

Active MQ configuration is based on PooledConnection and to limit the amount
of memory (Per-Destination) I set:
<policyEntry queue="IF_CBIKIT" producerFlowControl="true"
optimizedDispatch="true" reduceMemoryFootprint="true" memoryLimit="10Mb" >      
                                         
</policyEntry>

But, after about 1h the system goes in OutOfMemory.

Thanks you so much for your help.

Best Regards

Michele





 



--
View this message in context: 
http://camel.465427.n5.nabble.com/Best-Strategy-to-process-a-large-number-of-rows-in-File-tp5779856p5781003.html
Sent from the Camel - Users mailing list archive at Nabble.com.