On Sep 22, 7:48 pm, "Steve Chu" <[EMAIL PROTECTED]> wrote: > Yes, we can not say it does not block, but it runs fast:)
Heh. Fast is relative. :) Your tps reports will need to account for the number of times a job is requested and none is available. > > Does it ever drop queued items? > > No, it is persistent(we use Berkeley DB TDS in our backend). But for > '-N' mode, if the server daemon crashed, it will lose the data in the > write buffer(the size determined by you with '-L' option), seem we > should use replication for this situation, but a bit complicated. Ah, good. > > Can you compare it to gearmand/beanstalkd/starling? > > We need a queue component that we know all the details(I mean the code > base, and when something bad happened, we can solve it easily). We do > not use Perl, we do not use Ruby, so we do not use gearmand and > starling. The application developers in our company are very familiar > with memcached clients, so we take the memcache protocol as a > consideration. I use beanstalkd because it's a really nice and small C codebase. :) You should look at it before putting too much more effort into where you're going here, as it already solves a lot of the problems (priority queues, time delayed jobs, blocking and partially blocking (including non-blocking) job reservation, automatic job reclamation due to timeout or disconnect, per-job stats, all kinds of stuff). What it doesn't do yet is persistence. That probably matters less to my applications than to other people's (since I've never had it crash and I can afford to theoretically lose a few jobs at the moment). [more below] On Sep 22, 8:00 pm, "Steve Chu" <[EMAIL PROTECTED]> wrote: > You mean the non-blocked I/O model? No, I mean if the queue doesn't block on a take (at least optionally), then it can't perform well in the general case. Clients will have to spend a lot of time just polling the queue. That's a lot of wasted network and CPU resources on both ends. You want the server to just hang until a job becomes available, and then immediately return. > What properties that make it > quite poorly? Can you give us a bit more detail? The set and get semantics can make a really simple fifo queue you can poll...but what if you need a priority queue (I've needed it every time I've implemented a job queue so far)? The polling thing is similarly an issue. You can block, not block, or partially block (with timeout). You'll want to support these things, or you're in bad shape. The memcached protocol doesn't have any room for something like, say, a job identifier that can be used to uniquely address the job on the way in or out. You basically just have a queue and you're forced to do an atomic delete and get from the queue in order to get things to work (or maybe a get and delete which would at least allow you to ensure the bulk of the contents were delivered before deleting it. How do you deal with a queue with delay? That's necessary in *at least* retry scenarios, but I've needed it directly in a few projects as well. > > If it's stateless, then you also have to have semantics for jobs > > that get dropped by clients. e.g. if I ask for a job and then > > immediately crash, does the job get run? > > Yes, this is a problem we should solve. Maybe a visible timeout and > explicit delete will make this done, but also a bit more complicated > to end users. You're going to have a really hard time getting that to work with existing memcached clients. That's not what they do.
