I'm integrating proper asynchronous support into pybeanstalk, which
will be released shortly and can be followed here:
http://github.com/tjdziuba/pybeanstalk,
but I want to ask a question regarding workers, I refer to them as
clients, and their appropriate actions.
Assuming that there's a pool of beanstalk queues, distributed across
multiple servers, is it OK to assume that if a client broadcasts a
reserve command, then all servers will receive "reserve\r\n"? Wouldn't
*ALL* queues then receive a "reserve" command which will wait
indefinitely if there are no jobs on the queue, hindering subsequent
commands?
Imagine the following every-day scenario, where there are producers
and consumers running on different machines:
producer:
p = Pool([Server1, Server2, Server3, Server4])
p.put(RandomData) #only one job is placed, randomly on a server chosen
from Pool
consumer:
c = connect_to_pool()
while True:
job = c.reserve() #issue reserve to all queues
# ... do something with the job ...
print c.stats()
All queues, except the random recipient of the placed job, now have a
"reserve\r\n" command which hinders them from processing any more
commands, e.g. the "stats\r\n" command.
To solve this issue, I can assume that there will be more than one job
placed, and the process of randomly choosing a server is fair and will
really distribute across multiple servers. To avoid including the
waiting servers in my randomly chosen server, I have given to them a
"waiting" attribute where, if evaluated to true, it excludes them from
the random broadcast eligibility.
Another way, is pretty hacky, but solves the issue of small jobs, if
there are more than one server then implicitly convert all reserve\r\n
commands to have a default timeout of something small like 5, where I
can constantly poll all servers.
Thoughts? Comments?
Mahmoud
--
You received this message because you are subscribed to the Google Groups
"beanstalk-talk" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/beanstalk-talk?hl=en.
