Do not use marshal string, it reads the content into memory.
On Mon, Apr 18, 2016 at 8:15 AM, Ravindra Godbole <godbol...@gmail.com> wrote:
> Have u tried this route ?
>
> from("file://src/data?readLock=changed&readLockCheckInterval=1500").marshal().string("UTF-8")
> .split(body().tokenize("\n"))
> .streaming().setHeader("xxx", constant("xxx"))
> .aggregate(header("xxx"), new
> StringBodyAggregator()).completionSize(100).completionTimeout(1500)
> .process(new Processor() {
> public void process(Exchange exchange) throws Exception {
> // Process your batch here.... send it to kafka or whatever...
> }
>
> });
>
>
> On Mon, Apr 18, 2016 at 11:34 AM, h0mer <mad4c...@gmail.com> wrote:
>
>> That's sort of the problem. It seems I can't even do a
>> marshal().string("UTF-8") before the split as it seems to run out of heap
>> space.
>>
>> Here's what I've tried:
>>
>> public void configure() throws Exception {
>> from(ftpToKafkaObj.getFtpEndpoint())
>> .marshal().string("UTF-8")
>> .split(body().tokenize("\n"))
>> .streaming()
>> .process(new FtpToLocalFileProcessor())
>> .routeId("ROUTEID::::: " + ftpToKafkaObj.getRouteId())
>> .aggregate(constant(true),batchAggregationStrategy())
>> .completionPredicate(batchSizePredicate())
>> .to(ftpToKafkaObj.getKafkaEndpoint())
>> .end();
>> }
>>
>> Now the above also throws an OOME as the "marshal().string("UTF-8")" line
>> above tries to convert the entire object to a string.
>>
>> If I remove the "marshal().string("UTF-8")" line above, it goes through,
>> but
>> the exchange does not receive the content of the file, but rather just the
>> file name and some other information (like my original post above states).
>> Keep in mind that this is a 3gig file with about 7 million lines (apache
>> log
>> file).
>>
>>
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/Camel-Large-File-Processing-Issues-tp5781221p5781236.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> - Ravi
>
> *[ View My Youtube Channel ] <https://www.youtube.com/c/RavindraGodbole>*
>
> Phone: +91 *98 509 760 91*
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
