Oh, sorry. that's right, it is 64 MB. and I tried with 16, 32, 64 and 128 MB. I will try the buffer size option of the file-comp. to see if it solves the problem. Thanks, Farhad S.
Claus Ibsen-2 wrote: > > On Fri, Nov 27, 2009 at 2:26 PM, shirazi <m...@farhad.eu> wrote: >> >> That's probably the case. I've tried with different buffer sizes (though >> not >> 0), from 16 to 128 KB. I got the best performance with buffer of 64KB, on >> Linux Debian with 4 Gigs RAM, and files larger than 3 Gigs. >> That would be great, if one could specify the buffer-size for the "file" >> component. >> > > You buffer size was 64 * 1024 * 1024 which is 64 K * K = 64 MB. Or is > there something that play tricks on me? > > The file component already have a bufferSize option with a default of > 128kb > http://camel.apache.org/file2.html > > So you can use that to set what you want to use. > > > >> -Farhad S. >> >> >> Claus Ibsen-2 wrote: >>> >>> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <m...@farhad.eu> wrote: >>>> >>>> Sure. I have attached the file. >>>> Actually I use from->to components for transfering files and I guess it >>>> copies the file, rather than moving. >>>> >>> >>> Ah makes more sense for copying to leverage NIO. >>> >>> Camel does this also but I can see that it may hit that 2gb limit as >>> it doesnt transfer in a loop using a lower buffer. >>> >>> It was doing it in a one liner >>> in.transferTo(0, in.size(), out); >>> >>> I wonder if you really must do that in a loop to avoid any 2gb limit >>> or the likes? >>> >>> >>>> Thanks, >>>> Farhad S. >>>> http://old.nabble.com/file/p26542125/Nio.java Nio.java >>>> >>>> >>>> Claus Ibsen-2 wrote: >>>>> >>>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <m...@farhad.eu> wrote: >>>>>> >>>>>> Wau... thanks for that. I don't understand how I could miss it, cause >>>>>> i >>>>>> searched a lot for such a feature. >>>>>> >>>>>> Another thing that I have experienced in 2.0, is that the "file" >>>>>> component >>>>>> is unable to move/copy files larger than 2Gigs, such large files are >>>>>> cut >>>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with >>>>>> the >>>>>> OS >>>>>> or the filesystems or what. I have seen it both in Linux (debian) and >>>>>> Mac >>>>>> OS >>>>>> X. And I don't understand either, why the file component does not use >>>>>> the >>>>>> NIO api or at least we don't have an option to tell it use the NIO >>>>>> api >>>>>> instead. >>>>>> Anyway, for now I've solved the problem by using a smiple component >>>>>> instead >>>>>> of the "file" comp. to move/copy files using the NIO api. It is much >>>>>> faster >>>>>> and it has no problem with very large files. >>>>>> >>>>> >>>>> For move it uses java.io.File rename and are you saying its not >>>>> optimized for the underlying OS? >>>>> I would assume that it could move a 2gb file in no time if the file is >>>>> on the same share. >>>>> >>>>> Can you share your code that you are using? >>>>> >>>>> >>>>>> Bests, >>>>>> Farhad S. >>>>>> >>>>>> >>>>>> Claus Ibsen-2 wrote: >>>>>>> >>>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <m...@farhad.eu> wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> Ok, I guess I should rather provide an AggregationStrategy. >>>>>>>> The reason why I use a timer is that when I use the ftp endpoint, >>>>>>>> it >>>>>>>> starts >>>>>>>> to fetch everything that is on the ftp in the first call and that's >>>>>>>> not >>>>>>>> what >>>>>>>> i want, because I'm processing files over 3 Gigs and there are many >>>>>>>> of >>>>>>>> them. >>>>>>>> Then I figured that by using a timer, i could have ftp to download >>>>>>>> one >>>>>>>> file >>>>>>>> at a time!! >>>>>>>> What I basically need is to somehow tell the ftp, how many files I >>>>>>>> want >>>>>>>> to >>>>>>>> fetch at a time. This is probably doable now in camel-2.1 by using >>>>>>>> the >>>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet. >>>>>>>> Do you have any suggestions? >>>>>>> >>>>>>> Yeah the FTP component extends the File component so you got all the >>>>>>> options from the file component as well >>>>>>> http://camel.apache.org/file2.html >>>>>>> >>>>>>> And there is a maxMessagesPerPoll you can use to tell how many files >>>>>>> you want at most to poll. For example = 1 in your use. >>>>>>> >>>>>>> And that option is avail in 2.0 also. >>>>>>> >>>>>>> In fact all these are as they are batch polling consumers >>>>>>> http://camel.apache.org/batch-consumer.html >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Bests, >>>>>>>> Farhad S. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Claus Ibsen-2 wrote: >>>>>>>>> >>>>>>>>> Hi >>>>>>>>> >>>>>>>>> BTW why are you using a timer -> ftp when the ftp have a build in >>>>>>>>> timer? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <m...@farhad.eu> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> Please consider the following route: >>>>>>>>>> >>>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000"). >>>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams";, >>>>>>>>>> 10000). >>>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress"); >>>>>>>>>> >>>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there >>>>>>>>>> is >>>>>>>>>> nothing >>>>>>>>>> to fetch from the ftp. I got rid of the exception by a >>>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route. >>>>>>>>>> Now >>>>>>>>>> in >>>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to" >>>>>>>>>> endpoint: >>>>>>>>>> ================= >>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> --> >>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly, >>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009}, >>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>>>>>> Cannot store file: >>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> --> >>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly, >>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009}, >>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>>>>>> Cannot store file: >>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file: >>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>>>>>> Cannot >>>>>>>>>> store file: >>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>>>> at >>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50) >>>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512) >>>>>>>>>> at java.util.TimerThread.run(Timer.java:462) >>>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body >>>>>>>>>> available >>>>>>>>>> of >>>>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by: >>>>>>>>>> No >>>>>>>>>> type >>>>>>>>>> converter available to convert from type: null to the required >>>>>>>>>> type: >>>>>>>>>> java.io.InputStream with value null on the exchange: >>>>>>>>>> Exchange[Message: >>>>>>>>>> [Body >>>>>>>>>> is null]] >>>>>>>>>> at >>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198) >>>>>>>>>> ... 29 more >>>>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: >>>>>>>>>> No >>>>>>>>>> type >>>>>>>>>> converter available to convert from type: null to the required >>>>>>>>>> type: >>>>>>>>>> java.io.InputStream with value null >>>>>>>>>> at >>>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126) >>>>>>>>>> at >>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101) >>>>>>>>>> ... 31 more >>>>>>>>>> ============ >>>>>>>>>> Now, I can remedy the situation by a try-catch block or an >>>>>>>>>> interceptor, >>>>>>>>>> but >>>>>>>>>> in my opinion this should be considered a bug in camel and it >>>>>>>>>> should >>>>>>>>>> silently ignore if there was nothing to fetch from the ftp. >>>>>>>>>> >>>>>>>>>> Bests, >>>>>>>>>> Farhad Shiraz >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> View this message in context: >>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html >>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Claus Ibsen >>>>>>>>> Apache Camel Committer >>>>>>>>> >>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>>>>> Open Source Integration: http://fusesource.com >>>>>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>>>>> Twitter: http://twitter.com/davsclaus >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> View this message in context: >>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html >>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Claus Ibsen >>>>>>> Apache Camel Committer >>>>>>> >>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>>> Open Source Integration: http://fusesource.com >>>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>>> Twitter: http://twitter.com/davsclaus >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html >>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Claus Ibsen >>>>> Apache Camel Committer >>>>> >>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>> Open Source Integration: http://fusesource.com >>>>> Blog: http://davsclaus.blogspot.com/ >>>>> Twitter: http://twitter.com/davsclaus >>>>> >>>>> >>>> >>>> -- >>>> View this message in context: >>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html >>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>> >>>> >>> >>> >>> >>> -- >>> Claus Ibsen >>> Apache Camel Committer >>> >>> Author of Camel in Action: http://www.manning.com/ibsen/ >>> Open Source Integration: http://fusesource.com >>> Blog: http://davsclaus.blogspot.com/ >>> Twitter: http://twitter.com/davsclaus >>> >>> >> >> -- >> View this message in context: >> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542396.html >> Sent from the Camel - Users mailing list archive at Nabble.com. >> >> > > > > -- > Claus Ibsen > Apache Camel Committer > > Author of Camel in Action: http://www.manning.com/ibsen/ > Open Source Integration: http://fusesource.com > Blog: http://davsclaus.blogspot.com/ > Twitter: http://twitter.com/davsclaus > > -- View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542472.html Sent from the Camel - Users mailing list archive at Nabble.com.
