I'm not super familiar with the architecture of backgroundrb and i couldn't find the answer to this; are all items within a worker run serially? That is to say that if I'm using multiple methods within a worker, or even if different requests are enquiring calls to the same method, will these be able to run concurrently or are they going to delay each other to be serial?
I agree that I can sacrifice concurrency for memory and other resources (and deal with the sadness of having them all in the same file), but there are some background tasks, specifically the ones queued from a rails request, that i'd like to run concurrently somewhat so the response time isn't directly tied to how many people are online. \Peter From: Scott Windsor [mailto:[email protected]] Sent: Tuesday, August 11, 2009 10:54 AM To: hemant Cc: Peter Gengler; [email protected] Subject: Re: [Backgroundrb-devel] Question about backgroundrb and resource usage Shouldn't you be able to logically group these into separate workers to keep your code clean, but not have to pay the price of separate instances? While it's nice to have concurrency for each worker, it's not worth it (or feasible) memory-wise to most cases. Ideally, this would be a config setting to allow a set number workers to handle jobs. - scott On Mon, Aug 10, 2009 at 7:16 PM, hemant <[email protected]<mailto:[email protected]>> wrote: It totally depends on what you are doing. If your task is thread-safe you can even use thread_pool.defer() which will make sure that at least IO sensitive tasks are running concurrently. You don't need 10 workers there. One worker with some thread pool thrown around should suffice. On Tue, Aug 11, 2009 at 5:47 AM, Peter Gengler<[email protected]<mailto:[email protected]>> wrote: > You know that's an interesting point about the single worker; has anyone ever > used Skynet? Their fortay is distributing map/reduce problems in the > background, but they also mixed in a function to ActiveRecord called > send_later which allowed you to send off any method of a model for later > processing. I wrote my own mixin that does the same thing with Backgroundrb > and a GenericWorker class (and I added send_now_or_later so I could have a > fallback) because I liked keeping the functions on the model. That would > lend itself to reducing the backgroundrb overhead as you'd have a single > worker for any asynchronous AR processing, although it doesn't help with the > cron-style jobs. > > \Peter > > -----Original Message----- > From: Adam Williams [mailto:[email protected]<mailto:[email protected]>] > Sent: Monday, August 10, 2009 6:50 PM > To: Peter Gengler > Cc: [email protected]<mailto:[email protected]> > Subject: Re: [Backgroundrb-devel] Question about backgroundrb and resource > usage > > On Aug 10, 2009, at 3:44 PM, Peter Gengler wrote: > >> So I have a somewhat general question about backgroundrb and >> resource usage. I hadn't really considered the fact that it spools >> up a new rails process for each worker until I started making more >> workers; now that I'm have half a dozen it seems like the current >> design is really unscalable. I.e. if my rails stack takes 200MB of >> memory for example, that means 10 workers use 2GB of memory? That >> seems overly excessive to me, am I missing something that prevents >> that resource bloat? > > As far as I have seen, you're not missing anything. I had the same > problem and ended up having a single worker, with different methods > for the work. I'm considering getting rid of backgroundrb altogether, > favoring instead simply loading ActiveRecord and the model classes I > need, with the hope that a cron task can fire the thing up within a > few seconds. We need tasks to run every minute, so it must be fast > loading. Anyway, that would allow us to unload a couple hundred meg at > Engine Yard. > > adam > _______________________________________________ > Backgroundrb-devel mailing list > [email protected]<mailto:[email protected]> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org _______________________________________________ Backgroundrb-devel mailing list [email protected]<mailto:[email protected]> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
_______________________________________________ Backgroundrb-devel mailing list [email protected] http://rubyforge.org/mailman/listinfo/backgroundrb-devel
