Hi I have created a ticket about the NIO thingy https://issues.apache.org/activemq/browse/CAMEL-2234
On Fri, Nov 27, 2009 at 2:14 PM, Claus Ibsen <claus.ib...@gmail.com> wrote: > On Fri, Nov 27, 2009 at 2:00 PM, shirazi <m...@farhad.eu> wrote: >> >> Sure. I have attached the file. >> Actually I use from->to components for transfering files and I guess it >> copies the file, rather than moving. >> > > Ah makes more sense for copying to leverage NIO. > > Camel does this also but I can see that it may hit that 2gb limit as > it doesnt transfer in a loop using a lower buffer. > > It was doing it in a one liner > in.transferTo(0, in.size(), out); > > I wonder if you really must do that in a loop to avoid any 2gb limit > or the likes? > > >> Thanks, >> Farhad S. >> http://old.nabble.com/file/p26542125/Nio.java Nio.java >> >> >> Claus Ibsen-2 wrote: >>> >>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <m...@farhad.eu> wrote: >>>> >>>> Wau... thanks for that. I don't understand how I could miss it, cause i >>>> searched a lot for such a feature. >>>> >>>> Another thing that I have experienced in 2.0, is that the "file" >>>> component >>>> is unable to move/copy files larger than 2Gigs, such large files are cut >>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with the >>>> OS >>>> or the filesystems or what. I have seen it both in Linux (debian) and Mac >>>> OS >>>> X. And I don't understand either, why the file component does not use the >>>> NIO api or at least we don't have an option to tell it use the NIO api >>>> instead. >>>> Anyway, for now I've solved the problem by using a smiple component >>>> instead >>>> of the "file" comp. to move/copy files using the NIO api. It is much >>>> faster >>>> and it has no problem with very large files. >>>> >>> >>> For move it uses java.io.File rename and are you saying its not >>> optimized for the underlying OS? >>> I would assume that it could move a 2gb file in no time if the file is >>> on the same share. >>> >>> Can you share your code that you are using? >>> >>> >>>> Bests, >>>> Farhad S. >>>> >>>> >>>> Claus Ibsen-2 wrote: >>>>> >>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <m...@farhad.eu> wrote: >>>>>> >>>>>> Hi, >>>>>> Ok, I guess I should rather provide an AggregationStrategy. >>>>>> The reason why I use a timer is that when I use the ftp endpoint, it >>>>>> starts >>>>>> to fetch everything that is on the ftp in the first call and that's not >>>>>> what >>>>>> i want, because I'm processing files over 3 Gigs and there are many of >>>>>> them. >>>>>> Then I figured that by using a timer, i could have ftp to download one >>>>>> file >>>>>> at a time!! >>>>>> What I basically need is to somehow tell the ftp, how many files I want >>>>>> to >>>>>> fetch at a time. This is probably doable now in camel-2.1 by using the >>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet. >>>>>> Do you have any suggestions? >>>>> >>>>> Yeah the FTP component extends the File component so you got all the >>>>> options from the file component as well >>>>> http://camel.apache.org/file2.html >>>>> >>>>> And there is a maxMessagesPerPoll you can use to tell how many files >>>>> you want at most to poll. For example = 1 in your use. >>>>> >>>>> And that option is avail in 2.0 also. >>>>> >>>>> In fact all these are as they are batch polling consumers >>>>> http://camel.apache.org/batch-consumer.html >>>>> >>>>> >>>>> >>>>>> >>>>>> Bests, >>>>>> Farhad S. >>>>>> >>>>>> >>>>>> >>>>>> Claus Ibsen-2 wrote: >>>>>>> >>>>>>> Hi >>>>>>> >>>>>>> BTW why are you using a timer -> ftp when the ftp have a build in >>>>>>> timer? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <m...@farhad.eu> wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> Please consider the following route: >>>>>>>> >>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000"). >>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams";, >>>>>>>> 10000). >>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress"); >>>>>>>> >>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there is >>>>>>>> nothing >>>>>>>> to fetch from the ftp. I got rid of the exception by a >>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route. Now >>>>>>>> in >>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to" >>>>>>>> endpoint: >>>>>>>> ================= >>>>>>>> 12:37:06,899 INFO [Tracer] >>> --> >>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly, >>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009}, >>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>>>> Cannot store file: >>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>> 12:37:06,899 INFO [Tracer] >>> --> >>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly, >>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009}, >>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>>>> Cannot store file: >>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file: >>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>>>> Cannot >>>>>>>> store file: >>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>> at >>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204) >>>>>>>> at >>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184) >>>>>>>> at >>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126) >>>>>>>> at >>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57) >>>>>>>> at >>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97) >>>>>>>> at >>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95) >>>>>>>> at >>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146) >>>>>>>> at >>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94) >>>>>>>> at >>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82) >>>>>>>> at >>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>> at >>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>>>>>> at >>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>>>>>> at >>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162) >>>>>>>> at >>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>> at >>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206) >>>>>>>> at >>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74) >>>>>>>> at >>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63) >>>>>>>> at >>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>> at >>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>>>>>> at >>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>>>>>> at >>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162) >>>>>>>> at >>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>> at >>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206) >>>>>>>> at >>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54) >>>>>>>> at >>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48) >>>>>>>> at >>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>> at >>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103) >>>>>>>> at >>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50) >>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512) >>>>>>>> at java.util.TimerThread.run(Timer.java:462) >>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body >>>>>>>> available >>>>>>>> of >>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by: No >>>>>>>> type >>>>>>>> converter available to convert from type: null to the required type: >>>>>>>> java.io.InputStream with value null on the exchange: >>>>>>>> Exchange[Message: >>>>>>>> [Body >>>>>>>> is null]] >>>>>>>> at >>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103) >>>>>>>> at >>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116) >>>>>>>> at >>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198) >>>>>>>> ... 29 more >>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No >>>>>>>> type >>>>>>>> converter available to convert from type: null to the required type: >>>>>>>> java.io.InputStream with value null >>>>>>>> at >>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126) >>>>>>>> at >>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101) >>>>>>>> ... 31 more >>>>>>>> ============ >>>>>>>> Now, I can remedy the situation by a try-catch block or an >>>>>>>> interceptor, >>>>>>>> but >>>>>>>> in my opinion this should be considered a bug in camel and it should >>>>>>>> silently ignore if there was nothing to fetch from the ftp. >>>>>>>> >>>>>>>> Bests, >>>>>>>> Farhad Shiraz >>>>>>>> >>>>>>>> -- >>>>>>>> View this message in context: >>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html >>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Claus Ibsen >>>>>>> Apache Camel Committer >>>>>>> >>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>>> Open Source Integration: http://fusesource.com >>>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>>> Twitter: http://twitter.com/davsclaus >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html >>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Claus Ibsen >>>>> Apache Camel Committer >>>>> >>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>> Open Source Integration: http://fusesource.com >>>>> Blog: http://davsclaus.blogspot.com/ >>>>> Twitter: http://twitter.com/davsclaus >>>>> >>>>> >>>> >>>> -- >>>> View this message in context: >>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html >>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>> >>>> >>> >>> >>> >>> -- >>> Claus Ibsen >>> Apache Camel Committer >>> >>> Author of Camel in Action: http://www.manning.com/ibsen/ >>> Open Source Integration: http://fusesource.com >>> Blog: http://davsclaus.blogspot.com/ >>> Twitter: http://twitter.com/davsclaus >>> >>> >> >> -- >> View this message in context: >> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html >> Sent from the Camel - Users mailing list archive at Nabble.com. >> >> > > > > -- > Claus Ibsen > Apache Camel Committer > > Author of Camel in Action: http://www.manning.com/ibsen/ > Open Source Integration: http://fusesource.com > Blog: http://davsclaus.blogspot.com/ > Twitter: http://twitter.com/davsclaus > -- Claus Ibsen Apache Camel Committer Author of Camel in Action: http://www.manning.com/ibsen/ Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/ Twitter: http://twitter.com/davsclaus
