Thanks guys... that's a ton of info! I am definetely gonna use the thread_pool... as soon as I can find the documentation ;D
1- For each feed, I define a "frequency" (every minute, every hour... every 30 minutes...) that will be updated every time I'm parsing the feed: if the parser returns "new" element, I am increasding the frequency (from 1 time per hour, to 1 time per 30 min.), if not, I'm decreasing the frequency... 2- I also have a "last_update" field which remembers the time when the feed was parsed for the last time. 3- With 1 & 2, I know how "late" I am to parse a feed... so when I choose my next feed to parse, I am always choosing the one that is the most "late" I am not sure if Steevie's approach of having multiple tasks for the worker applies here. Actually, I am not even schedulling my worker, I am just launching it once, and the parse_feeds runs forever (while true do... end) Also, if I understand well Paul's code, his approach allows my worker to be more efficient always, but doesn't take into account the "lateness" of my feeds. My idea would be to add/remove worker according to "how late" I am in parsing feeds. If my the the lastest feed is late by more than 10min, I would add one worker... and If my latest feed is late by less than 5 minutes, I would remove one worker Does this approach makes sense to you? Thanks a lot for your help guys... On 4/23/08, Paul Kmiec <[EMAIL PROTECTED]> wrote: > You can use the built build thread pool to process more than one feed within > the same worker. So within the worker, you'd do, > > def parse_feeds > loop do > feed = Feed.find_feed_to_process > thread_pool.defer do > feed.parse > end > end > end > > I think the default pool size is 20. You can control the size of the thread > pool using a class level method, as I recall it is > > pool_size x > > Paul > > > On Wed, Apr 23, 2008 at 7:30 AM, Julien Genestoux > <[EMAIL PROTECTED]> wrote: > > Thanks Adam, > > > > That sounded weird to me as well to have one worker for each feed... > > However, if I only have one worker, that also means that I am parsing > > one feed only at any moment. An option, is maybe to have a few workers > > (denpending on the number of feeds) that parse feeds concurrently? > > > > If I only have one worker, according to you what should be the > > winnning strategy to choose the "right" parse to feed? Obviously some > > feeds need to be parsed one every few minutes, while some other might > > no need to be parse more than every hour... > > > > Any idea/tip on this? > > > > > > > > > > > > > > > > On 4/23/08, Adam Williams <[EMAIL PROTECTED]> wrote: > > > On Apr 23, 2008, at 1:07 AM, Julien Genestoux wrote: > > > > > > > I still have a few questions : shoud I have one worker for each feed > > > > that is called periodically (add_periodic_timer) or rather one single > > > > worker that calls every feed one by one? > > > > > > > > What is the best solution, perfomance-wise? > > > > > > > > > Good question... I don't suppose I know exactly. I would start by > > > processing all the feeds in one worker invocation - that is what I > > > have done for sending an unknown amount of email. It just seems wrong > > > to me to invoke a worker for one email at a time. > > > > > > The right answer likely lies in understanding the whole MasterWorker, > > > Packet::Reactor/handler_instance.ask_work bits of the > puzzle... > > > > > > > > > adam > > > > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > [email protected] > > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > > > -- > > -- > > Julien Genestoux > > [EMAIL PROTECTED] > > http://www.ouvre-boite.com > > +1 (415) 254 7340 > > +33 (0)8 70 44 76 29 > > _______________________________________________ > > > > > > > > Backgroundrb-devel mailing list > > [email protected] > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > -- -- Julien Genestoux [EMAIL PROTECTED] http://www.ouvre-boite.com +1 (415) 254 7340 +33 (0)8 70 44 76 29 _______________________________________________ Backgroundrb-devel mailing list [email protected] http://rubyforge.org/mailman/listinfo/backgroundrb-devel
