I considered decreasing the poll interval as well, but couldn't get cpu usage down enough for acceptable interval...
I'm trying PollingConsumerPollStrategy but begin method is a void not a boolean, isn't it? Claus Ibsen-2 wrote: > > On Mon, Feb 22, 2010 at 9:04 AM, wmoussel <[email protected]> wrote: >> >> Hi, >> >> Here's my initial issue. I have to poll for new files in a directory >> containing a lot of files (5000 to 20000) (and I can't move them after >> processing). >> >> When using idempotent option it takes way too much cpu (like 9% even when >> they aren't new files). >> > > How often are you gonna poll in those files? You can probably change > the delay to be less frequent. > > In any case somehow you gotta match all those 20000 files whether or > not you have processed them before. > > >> So i tried adding a filter so that the consumer would only take files >> less >> than 2 days old. It doesn't change much in the end. When I looked into >> how >> filter works in Java I got why it's still cpu consuming... >> >> So then I thought i would get recent file names with a unix command like >> find . -mtime -1 and use pollenrich to poll specific files. The way I >> understand how poll enrich works, it will poll the whole directory and >> then >> try to match filename afterwards. That is also cpu consuming since it >> gets >> all the files as exchanges prior to the match. Am I wrong ? >> > > pollEnrich is not suitable for this as its meant for polling and > aggregating 1 resource at a time. > >> So I'm basically trying to invoke a FileName to GenericFileExchange >> processor in the middle of my route. Is there a way to do this? >> >> I'm open to any suggestion. >> > > The file component supports using a custom PollingConsumerPollStrategy > http://camel.apache.org/polling-consumer.html > > You can then implement your own logic and return false in the being > method where there are no new files since last time. > > >> Thanks in advance :) >> >> Wandrille >> -- >> View this message in context: >> http://old.nabble.com/File-Consumer-with-dynamic-list-of-files-to-poll-tp27683938p27683938.html >> Sent from the Camel - Users mailing list archive at Nabble.com. >> >> > > > > -- > Claus Ibsen > Apache Camel Committer > > Author of Camel in Action: http://www.manning.com/ibsen/ > Open Source Integration: http://fusesource.com > Blog: http://davsclaus.blogspot.com/ > Twitter: http://twitter.com/davsclaus > > -- View this message in context: http://old.nabble.com/File-Consumer-with-dynamic-list-of-files-to-poll-tp27683938p27686162.html Sent from the Camel - Users mailing list archive at Nabble.com.
