On Fri, Nov 27, 2009 at 2:34 PM, shirazi <m...@farhad.eu> wrote: > > Oh, sorry. that's right, it is 64 MB. and I tried with 16, 32, 64 and 128 MB. > I will try the buffer size option of the file-comp. to see if it solves the > problem. >
I have just committed the fix to camel trunk. The old code will still have the 2gb limit bug no matter what bufferSize you use. However you should try the latest code in 2.1 to see if that works for you. http://camel.apache.org/download.html > Thanks, > Farhad S. > > > Claus Ibsen-2 wrote: >> >> On Fri, Nov 27, 2009 at 2:26 PM, shirazi <m...@farhad.eu> wrote: >>> >>> That's probably the case. I've tried with different buffer sizes (though >>> not >>> 0), from 16 to 128 KB. I got the best performance with buffer of 64KB, on >>> Linux Debian with 4 Gigs RAM, and files larger than 3 Gigs. >>> That would be great, if one could specify the buffer-size for the "file" >>> component. >>> >> >> You buffer size was 64 * 1024 * 1024 which is 64 K * K = 64 MB. Or is >> there something that play tricks on me? >> >> The file component already have a bufferSize option with a default of >> 128kb >> http://camel.apache.org/file2.html >> >> So you can use that to set what you want to use. >> >> >> >>> -Farhad S. >>> >>> >>> Claus Ibsen-2 wrote: >>>> >>>> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <m...@farhad.eu> wrote: >>>>> >>>>> Sure. I have attached the file. >>>>> Actually I use from->to components for transfering files and I guess it >>>>> copies the file, rather than moving. >>>>> >>>> >>>> Ah makes more sense for copying to leverage NIO. >>>> >>>> Camel does this also but I can see that it may hit that 2gb limit as >>>> it doesnt transfer in a loop using a lower buffer. >>>> >>>> It was doing it in a one liner >>>> in.transferTo(0, in.size(), out); >>>> >>>> I wonder if you really must do that in a loop to avoid any 2gb limit >>>> or the likes? >>>> >>>> >>>>> Thanks, >>>>> Farhad S. >>>>> http://old.nabble.com/file/p26542125/Nio.java Nio.java >>>>> >>>>> >>>>> Claus Ibsen-2 wrote: >>>>>> >>>>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <m...@farhad.eu> wrote: >>>>>>> >>>>>>> Wau... thanks for that. I don't understand how I could miss it, cause >>>>>>> i >>>>>>> searched a lot for such a feature. >>>>>>> >>>>>>> Another thing that I have experienced in 2.0, is that the "file" >>>>>>> component >>>>>>> is unable to move/copy files larger than 2Gigs, such large files are >>>>>>> cut >>>>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with >>>>>>> the >>>>>>> OS >>>>>>> or the filesystems or what. I have seen it both in Linux (debian) and >>>>>>> Mac >>>>>>> OS >>>>>>> X. And I don't understand either, why the file component does not use >>>>>>> the >>>>>>> NIO api or at least we don't have an option to tell it use the NIO >>>>>>> api >>>>>>> instead. >>>>>>> Anyway, for now I've solved the problem by using a smiple component >>>>>>> instead >>>>>>> of the "file" comp. to move/copy files using the NIO api. It is much >>>>>>> faster >>>>>>> and it has no problem with very large files. >>>>>>> >>>>>> >>>>>> For move it uses java.io.File rename and are you saying its not >>>>>> optimized for the underlying OS? >>>>>> I would assume that it could move a 2gb file in no time if the file is >>>>>> on the same share. >>>>>> >>>>>> Can you share your code that you are using? >>>>>> >>>>>> >>>>>>> Bests, >>>>>>> Farhad S. >>>>>>> >>>>>>> >>>>>>> Claus Ibsen-2 wrote: >>>>>>>> >>>>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <m...@farhad.eu> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> Ok, I guess I should rather provide an AggregationStrategy. >>>>>>>>> The reason why I use a timer is that when I use the ftp endpoint, >>>>>>>>> it >>>>>>>>> starts >>>>>>>>> to fetch everything that is on the ftp in the first call and that's >>>>>>>>> not >>>>>>>>> what >>>>>>>>> i want, because I'm processing files over 3 Gigs and there are many >>>>>>>>> of >>>>>>>>> them. >>>>>>>>> Then I figured that by using a timer, i could have ftp to download >>>>>>>>> one >>>>>>>>> file >>>>>>>>> at a time!! >>>>>>>>> What I basically need is to somehow tell the ftp, how many files I >>>>>>>>> want >>>>>>>>> to >>>>>>>>> fetch at a time. This is probably doable now in camel-2.1 by using >>>>>>>>> the >>>>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet. >>>>>>>>> Do you have any suggestions? >>>>>>>> >>>>>>>> Yeah the FTP component extends the File component so you got all the >>>>>>>> options from the file component as well >>>>>>>> http://camel.apache.org/file2.html >>>>>>>> >>>>>>>> And there is a maxMessagesPerPoll you can use to tell how many files >>>>>>>> you want at most to poll. For example = 1 in your use. >>>>>>>> >>>>>>>> And that option is avail in 2.0 also. >>>>>>>> >>>>>>>> In fact all these are as they are batch polling consumers >>>>>>>> http://camel.apache.org/batch-consumer.html >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Bests, >>>>>>>>> Farhad S. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Claus Ibsen-2 wrote: >>>>>>>>>> >>>>>>>>>> Hi >>>>>>>>>> >>>>>>>>>> BTW why are you using a timer -> ftp when the ftp have a build in >>>>>>>>>> timer? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <m...@farhad.eu> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> Please consider the following route: >>>>>>>>>>> >>>>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000"). >>>>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams";, >>>>>>>>>>> 10000). >>>>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress"); >>>>>>>>>>> >>>>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there >>>>>>>>>>> is >>>>>>>>>>> nothing >>>>>>>>>>> to fetch from the ftp. I got rid of the exception by a >>>>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route. >>>>>>>>>>> Now >>>>>>>>>>> in >>>>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to" >>>>>>>>>>> endpoint: >>>>>>>>>>> ================= >>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> --> >>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly, >>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009}, >>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>>>>>>> Cannot store file: >>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> --> >>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly, >>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009}, >>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>>>>>>> Cannot store file: >>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file: >>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException: >>>>>>>>>>> Cannot >>>>>>>>>>> store file: >>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075 >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50) >>>>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512) >>>>>>>>>>> at java.util.TimerThread.run(Timer.java:462) >>>>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body >>>>>>>>>>> available >>>>>>>>>>> of >>>>>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by: >>>>>>>>>>> No >>>>>>>>>>> type >>>>>>>>>>> converter available to convert from type: null to the required >>>>>>>>>>> type: >>>>>>>>>>> java.io.InputStream with value null on the exchange: >>>>>>>>>>> Exchange[Message: >>>>>>>>>>> [Body >>>>>>>>>>> is null]] >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198) >>>>>>>>>>> ... 29 more >>>>>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: >>>>>>>>>>> No >>>>>>>>>>> type >>>>>>>>>>> converter available to convert from type: null to the required >>>>>>>>>>> type: >>>>>>>>>>> java.io.InputStream with value null >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126) >>>>>>>>>>> at >>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101) >>>>>>>>>>> ... 31 more >>>>>>>>>>> ============ >>>>>>>>>>> Now, I can remedy the situation by a try-catch block or an >>>>>>>>>>> interceptor, >>>>>>>>>>> but >>>>>>>>>>> in my opinion this should be considered a bug in camel and it >>>>>>>>>>> should >>>>>>>>>>> silently ignore if there was nothing to fetch from the ftp. >>>>>>>>>>> >>>>>>>>>>> Bests, >>>>>>>>>>> Farhad Shiraz >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> View this message in context: >>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html >>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Claus Ibsen >>>>>>>>>> Apache Camel Committer >>>>>>>>>> >>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>>>>>> Open Source Integration: http://fusesource.com >>>>>>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>>>>>> Twitter: http://twitter.com/davsclaus >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> View this message in context: >>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html >>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Claus Ibsen >>>>>>>> Apache Camel Committer >>>>>>>> >>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>>>> Open Source Integration: http://fusesource.com >>>>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>>>> Twitter: http://twitter.com/davsclaus >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> View this message in context: >>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html >>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Claus Ibsen >>>>>> Apache Camel Committer >>>>>> >>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>> Open Source Integration: http://fusesource.com >>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>> Twitter: http://twitter.com/davsclaus >>>>>> >>>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html >>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Claus Ibsen >>>> Apache Camel Committer >>>> >>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>> Open Source Integration: http://fusesource.com >>>> Blog: http://davsclaus.blogspot.com/ >>>> Twitter: http://twitter.com/davsclaus >>>> >>>> >>> >>> -- >>> View this message in context: >>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542396.html >>> Sent from the Camel - Users mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> Claus Ibsen >> Apache Camel Committer >> >> Author of Camel in Action: http://www.manning.com/ibsen/ >> Open Source Integration: http://fusesource.com >> Blog: http://davsclaus.blogspot.com/ >> Twitter: http://twitter.com/davsclaus >> >> > > -- > View this message in context: > http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542472.html > Sent from the Camel - Users mailing list archive at Nabble.com. > > -- Claus Ibsen Apache Camel Committer Author of Camel in Action: http://www.manning.com/ibsen/ Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/ Twitter: http://twitter.com/davsclaus
