On Fri, Nov 27, 2009 at 2:00 PM, shirazi <m...@farhad.eu> wrote:
>
> Sure. I have attached the file.
> Actually I use from->to components for transfering files and I guess it
> copies the file, rather than moving.
>
Ah makes more sense for copying to leverage NIO.
Camel does this also but I can see that it may hit that 2gb limit as
it doesnt transfer in a loop using a lower buffer.
It was doing it in a one liner
in.transferTo(0, in.size(), out);
I wonder if you really must do that in a loop to avoid any 2gb limit
or the likes?
> Thanks,
> Farhad S.
> http://old.nabble.com/file/p26542125/Nio.java Nio.java
>
>
> Claus Ibsen-2 wrote:
>>
>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <m...@farhad.eu> wrote:
>>>
>>> Wau... thanks for that. I don't understand how I could miss it, cause i
>>> searched a lot for such a feature.
>>>
>>> Another thing that I have experienced in 2.0, is that the "file"
>>> component
>>> is unable to move/copy files larger than 2Gigs, such large files are cut
>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with the
>>> OS
>>> or the filesystems or what. I have seen it both in Linux (debian) and Mac
>>> OS
>>> X. And I don't understand either, why the file component does not use the
>>> NIO api or at least we don't have an option to tell it use the NIO api
>>> instead.
>>> Anyway, for now I've solved the problem by using a smiple component
>>> instead
>>> of the "file" comp. to move/copy files using the NIO api. It is much
>>> faster
>>> and it has no problem with very large files.
>>>
>>
>> For move it uses java.io.File rename and are you saying its not
>> optimized for the underlying OS?
>> I would assume that it could move a 2gb file in no time if the file is
>> on the same share.
>>
>> Can you share your code that you are using?
>>
>>
>>> Bests,
>>> Farhad S.
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <m...@farhad.eu> wrote:
>>>>>
>>>>> Hi,
>>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>>> The reason why I use a timer is that when I use the ftp endpoint, it
>>>>> starts
>>>>> to fetch everything that is on the ftp in the first call and that's not
>>>>> what
>>>>> i want, because I'm processing files over 3 Gigs and there are many of
>>>>> them.
>>>>> Then I figured that by using a timer, i could have ftp to download one
>>>>> file
>>>>> at a time!!
>>>>> What I basically need is to somehow tell the ftp, how many files I want
>>>>> to
>>>>> fetch at a time. This is probably doable now in camel-2.1 by using the
>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>>> Do you have any suggestions?
>>>>
>>>> Yeah the FTP component extends the File component so you got all the
>>>> options from the file component as well
>>>> http://camel.apache.org/file2.html
>>>>
>>>> And there is a maxMessagesPerPoll you can use to tell how many files
>>>> you want at most to poll. For example = 1 in your use.
>>>>
>>>> And that option is avail in 2.0 also.
>>>>
>>>> In fact all these are as they are batch polling consumers
>>>> http://camel.apache.org/batch-consumer.html
>>>>
>>>>
>>>>
>>>>>
>>>>> Bests,
>>>>> Farhad S.
>>>>>
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> BTW why are you using a timer -> ftp when the ftp have a build in
>>>>>> timer?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <m...@farhad.eu> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> Please consider the following route:
>>>>>>>
>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams";,
>>>>>>> 10000).
>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>>
>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there is
>>>>>>> nothing
>>>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route. Now
>>>>>>> in
>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to"
>>>>>>> endpoint:
>>>>>>> =================
>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>> Cannot store file:
>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>> Cannot store file:
>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>> Cannot
>>>>>>> store file:
>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>> at
>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>>> at
>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>>> at
>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>>> at
>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>>> at
>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>>> at
>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>>> at
>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>>> at
>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>>> at
>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>>> at
>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>> at
>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>> at
>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>> at
>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>>> at
>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>>> at
>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>> at
>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>> at
>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>> at
>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>>> at
>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>> at
>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>>> at
>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body
>>>>>>> available
>>>>>>> of
>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by: No
>>>>>>> type
>>>>>>> converter available to convert from type: null to the required type:
>>>>>>> java.io.InputStream with value null on the exchange:
>>>>>>> Exchange[Message:
>>>>>>> [Body
>>>>>>> is null]]
>>>>>>> at
>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>>> at
>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>>> at
>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>>> ... 29 more
>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No
>>>>>>> type
>>>>>>> converter available to convert from type: null to the required type:
>>>>>>> java.io.InputStream with value null
>>>>>>> at
>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>>> at
>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>>> ... 31 more
>>>>>>> ============
>>>>>>> Now, I can remedy the situation by a try-catch block or an
>>>>>>> interceptor,
>>>>>>> but
>>>>>>> in my opinion this should be considered a bug in camel and it should
>>>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>>>
>>>>>>> Bests,
>>>>>>> Farhad Shiraz
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context:
> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
