On Fri, Nov 27, 2009 at 2:26 PM, shirazi <m...@farhad.eu> wrote: > > That's probably the case. I've tried with different buffer sizes (though not > 0), from 16 to 128 KB. I got the best performance with buffer of 64KB, on > Linux Debian with 4 Gigs RAM, and files larger than 3 Gigs. > That would be great, if one could specify the buffer-size for the "file" > component. >
You buffer size was 64 * 1024 * 1024 which is 64 K * K = 64 MB. Or is there something that play tricks on me? The file component already have a bufferSize option with a default of 128kb http://camel.apache.org/file2.html So you can use that to set what you want to use. > -Farhad S. > > > Claus Ibsen-2 wrote: >> >> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <m...@farhad.eu> wrote: >>> >>> Sure. I have attached the file. >>> Actually I use from->to components for transfering files and I guess it >>> copies the file, rather than moving. >>> >> >> Ah makes more sense for copying to leverage NIO. >> >> Camel does this also but I can see that it may hit that 2gb limit as >> it doesnt transfer in a loop using a lower buffer. >> >> It was doing it in a one liner >> in.transferTo(0, in.size(), out); >> >> I wonder if you really must do that in a loop to avoid any 2gb limit >> or the likes? >> >> >>> Thanks, >>> Farhad S. >>> http://old.nabble.com/file/p26542125/Nio.java Nio.java >>> >>> >>> Claus Ibsen-2 wrote: >>>> >>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <m...@farhad.eu> wrote: >>>>> >>>>> Wau... thanks for that. I don't understand how I could miss it, cause i >>>>> searched a lot for such a feature. >>>>> >>>>> Another thing that I have experienced in 2.0, is that the "file" >>>>> component >>>>> is unable to move/copy files larger than 2Gigs, such large files are >>>>> cut >>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with >>>>> the >>>>> OS >>>>> or the filesystems or what. I have seen it both in Linux (debian) and >>>>> Mac >>>>> OS >>>>> X. And I don't understand either, why the file component does not use >>>>> the >>>>> NIO api or at least we don't have an option to tell it use the NIO api >>>>> instead. >>>>> Anyway, for now I've solved the problem by using a smiple component >>>>> instead >>>>> of the "file" comp. to move/copy files using the NIO api. It is much >>>>> faster >>>>> and it has no problem with very large files. >>>>> >>>> >>>> For move it uses java.io.File rename and are you saying its not >>>> optimized for the underlying OS? >>>> I would assume that it could move a 2gb file in no time if the file is >>>> on the same share. >>>> >>>> Can you share your code that you are using? >>>> >>>> >>>>> Bests, >>>>> Farhad S. >>>>> >>>>> >>>>> Claus Ibsen-2 wrote: >>>>>> >>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <m...@farhad.eu> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> Ok, I guess I should rather provide an AggregationStrategy. >>>>>>> The reason why I use a timer is that when I use the ftp endpoint, it >>>>>>> starts >>>>>>> to fetch everything that is on the ftp in the first call and that's >>>>>>> not >>>>>>> what >>>>>>> i want, because I'm processing files over 3 Gigs and there are many >>>>>>> of >>>>>>> them. >>>>>>> Then I figured that by using a timer, i could have ftp to download >>>>>>> one >>>>>>> file >>>>>>> at a time!! >>>>>>> What I basically need is to somehow tell the ftp, how many files I >>>>>>> want >>>>>>> to >>>>>>> fetch at a time. This is probably doable now in camel-2.1 by using >>>>>>> the >>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet. >>>>>>> Do you have any suggestions? >>>>>> >>>>>> Yeah the FTP component extends the File component so you got all the >>>>>> options from the file component as well >>>>>> http://camel.apache.org/file2.html >>>>>> >>>>>> And there is a maxMessagesPerPoll you can use to tell how many files >>>>>> you want at most to poll. For example = 1 in your use. >>>>>> >>>>>> And that option is avail in 2.0 also. >>>>>> >>>>>> In fact all these are as they are batch polling consumers >>>>>> http://camel.apache.org/batch-consumer.html >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Bests, >>>>>>> Farhad S. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Claus Ibsen-2 wrote: >>>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> BTW why are you using a timer -> ftp when the ftp have a build in >>>>>>>> timer? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <m...@farhad.eu> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> Please consider the following route: >>>>>>>>> >>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000"). >>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams";, >>>>>>>>> 10000). >>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress"); >>>>>>>>> >>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there >>>>>>>>> is >>>>>>>>> nothing >>>>>>>>> to fetch from the ftp. I got rid of the exception by a >>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route. >>>>>>>>> Now >>>>>>>>> in >>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to" >>>>>>>>> endpoint: >>>>>>>>> ================= >>>>>>>>> 12:37:06,899 INFO [Tracer] >>> --> >>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly, >>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009}, >>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>>>>> Cannot store file: >>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>>> 12:37:06,899 INFO [Tracer] >>> --> >>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly, >>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009}, >>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>>>>> Cannot store file: >>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file: >>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>>>>> Cannot >>>>>>>>> store file: >>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>>> at >>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204) >>>>>>>>> at >>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184) >>>>>>>>> at >>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126) >>>>>>>>> at >>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95) >>>>>>>>> at >>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82) >>>>>>>>> at >>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162) >>>>>>>>> at >>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63) >>>>>>>>> at >>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162) >>>>>>>>> at >>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54) >>>>>>>>> at >>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48) >>>>>>>>> at >>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>> at >>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103) >>>>>>>>> at >>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50) >>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512) >>>>>>>>> at java.util.TimerThread.run(Timer.java:462) >>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body >>>>>>>>> available >>>>>>>>> of >>>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by: >>>>>>>>> No >>>>>>>>> type >>>>>>>>> converter available to convert from type: null to the required >>>>>>>>> type: >>>>>>>>> java.io.InputStream with value null on the exchange: >>>>>>>>> Exchange[Message: >>>>>>>>> [Body >>>>>>>>> is null]] >>>>>>>>> at >>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103) >>>>>>>>> at >>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116) >>>>>>>>> at >>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198) >>>>>>>>> ... 29 more >>>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No >>>>>>>>> type >>>>>>>>> converter available to convert from type: null to the required >>>>>>>>> type: >>>>>>>>> java.io.InputStream with value null >>>>>>>>> at >>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126) >>>>>>>>> at >>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101) >>>>>>>>> ... 31 more >>>>>>>>> ============ >>>>>>>>> Now, I can remedy the situation by a try-catch block or an >>>>>>>>> interceptor, >>>>>>>>> but >>>>>>>>> in my opinion this should be considered a bug in camel and it >>>>>>>>> should >>>>>>>>> silently ignore if there was nothing to fetch from the ftp. >>>>>>>>> >>>>>>>>> Bests, >>>>>>>>> Farhad Shiraz >>>>>>>>> >>>>>>>>> -- >>>>>>>>> View this message in context: >>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html >>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Claus Ibsen >>>>>>>> Apache Camel Committer >>>>>>>> >>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>>>> Open Source Integration: http://fusesource.com >>>>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>>>> Twitter: http://twitter.com/davsclaus >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> View this message in context: >>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html >>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Claus Ibsen >>>>>> Apache Camel Committer >>>>>> >>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>> Open Source Integration: http://fusesource.com >>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>> Twitter: http://twitter.com/davsclaus >>>>>> >>>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html >>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Claus Ibsen >>>> Apache Camel Committer >>>> >>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>> Open Source Integration: http://fusesource.com >>>> Blog: http://davsclaus.blogspot.com/ >>>> Twitter: http://twitter.com/davsclaus >>>> >>>> >>> >>> -- >>> View this message in context: >>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html >>> Sent from the Camel - Users mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> Claus Ibsen >> Apache Camel Committer >> >> Author of Camel in Action: http://www.manning.com/ibsen/ >> Open Source Integration: http://fusesource.com >> Blog: http://davsclaus.blogspot.com/ >> Twitter: http://twitter.com/davsclaus >> >> > > -- > View this message in context: > http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542396.html > Sent from the Camel - Users mailing list archive at Nabble.com. > > -- Claus Ibsen Apache Camel Committer Author of Camel in Action: http://www.manning.com/ibsen/ Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/ Twitter: http://twitter.com/davsclaus
