I've been having problems with losing async'ed tasks, so I've moved to
trying to use enqueued tasks.
I'm trying to offload user queries to the background, using a thread
pool size of 40 and two workers (on different machines). It seems that
if I exceed the thread pool size those queries get lost, or for some
other reason I can't determine yet I'm losing my tasks when there are
lots of them at once.
For that reason I've moved to trying enqued tasks. This method seems
to be fine for not losing jobs but I can't seem to run more then 2 at
a time? (one for each worker). Is there a way to up the number of
enqueued tasks that will be ran at a time? This way I could make use
of my thread pool, and the enqueueing tasks.
The behavior I expected from this method is that it checks every
second (like I have configured) for enqueued tasks. Then running as
many enqueued tasks as the thread pool limit for that worker. Is there
a way to get this behavior from the system? Right now it seems to pull
out one at a time and run them till they finish and then pull out the
next.
If my expected behavior isn't how it's supposed to work, would there
be a way to do that?
I'm running version 1.0.4 with packet 0.1.14
My drb config:
---
:backgroundrb:
:ip: 0.0.0.0
:port: 11006
:persistent_delay: 1
:debug_log: true
:environment: development
:result_storage: memcache
:memcache: "xxx.xxx.xxx.xxx:11211,xxx.xxx.xxx.xxx:11211"
:client: "xxx.xxx.xxx.xxx:11006,xxx.xxx.xxx.xxx:11006"
# :schedules: # optional task scheduling
My worker:
require 'hpricot'
require 'open-uri'
class TexisWorker < BackgrounDRb::MetaWorker
set_worker_name :texis_worker
pool_size 40
def create(args = nil)
logger.info "Creating Texis Worker Instance"
end
def search(arg)
thread_pool.defer(:perform_search, arg)
end
def perform_search(arg)
# calls out to a server runs a search, gets parsed by hpricot and
then caches the results
persistent_job.finish!
end
end
Thanks,
Curtis
_______________________________________________
Backgroundrb-devel mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/backgroundrb-devel
