You are basically proposing the schema for DelayedJobs (Ruby) [ https://github.com/collectiveidea/delayed_job ]. This kind of thing gets really weird in eventually consistant databases, but Mongo (like mySQL) is always consistant so you should be OK. However, a lot of folks have been finding some locking problems with this approach (multiple job execution or really aggressive table locking is needed), so most folks tend to use a store which can support atomic "push" and "pop" operations so you can ensure that one and only one worker gets the job. Redis is the most popular of these types of stores these day. I make use of that property in http://actionherojs.com/ for exactly this purpose, as does the very popular https://github.com/defunkt/resque and some other projects.
On Friday, October 5, 2012 12:32:39 PM UTC-7, Mark Hahn wrote: > > I may be crazy but I'm implementing a scheme where processes get the tasks > from a db record and then stores their process number in that record. Then > while they are running they periodically check to make sure that their > server number is still the one in the record. If another process has > 'stolen' the task then the process aborts and looks for another one to do. > > This is the only way I could figure out how to do task assignment when > faced with a db that only has "eventual consistency". The CouchDB i'm > using offers no atomic operations so this was the only reliable way to do > it. > > It works quite well. In the usual case it just grabs the task, does it, > and moves on. Collisions are rare, but they may be more frequent as the > cluster grows in size. > > > On Fri, Oct 5, 2012 at 9:21 AM, Dan Milon <danm...@gmail.com <javascript:> > > wrote: > >> greelkorke ment using a job queue where jobs are put, and handed to >> workers. >> >> If you want to do it only with mongo, you'll need to use some "lock" >> document, that is set and unset by the first process which tries to >> initiate a task. All other processes which try to grab the lock while its >> held by another process should assume that the job is being worked by >> another process. But thats really ugly & has problems because jobs cant be >> acknowledged, so if a process crashes while its performing some task, >> you're fucked. >> >> On Fri, Oct 5, 2012 at 5:07 PM, Tom <tomm...@gmail.com <javascript:>>wrote: >> >>> Unfortunately I'm afraid that I don't see how a scheduler can avoid the >>> concurrency problems. Note that the advantages (e.g. in availability) of >>> having a cluster should be maintained here, and so you cannot run a >>> scheduler in a separate process on a single server. If every server would >>> be running the scheduler, the same concurrency problems would arise. What >>> were you proposing? >>> >>> About the initialization, I guess that would work. It is not the way I >>> would prefer to do it, as I would like the application to be >>> self-controlled and usable without running special tools, but I reckon it >>> is an acceptable approach. >>> >>> Tom >>> >>> Op vrijdag 5 oktober 2012 19:59:33 UTC+7 schreef greelgorke het volgende: >>> >>>> i wouldn't do it that way. when deploying your app just do a pre-start >>>> script, that ensures the existence of your desired data. >>>> If you have periodical tasks, it's best to use a lib for it, that >>>> triggers jobs appart of your main application. i.E >>>> http://stackoverflow.com/**questions/3785736/is-there-a-** >>>> job-scheduler-library-for-**node-js<http://stackoverflow.com/questions/3785736/is-there-a-job-scheduler-library-for-node-js>, >>>> so you just avoid the concurrency problems. >>>> >>>> Am Freitag, 5. Oktober 2012 14:04:58 UTC+2 schrieb Tom: >>>>> >>>>> I've setup a cluster of physical servers. Each server runs exactly the >>>>> same code. Moreover, each server runs multiple node processes using >>>>> the build in cluster functionality. >>>>> >>>>> I use MongoDB (native) to share information between processes and >>>>> servers. However, I am having some difficulty with running a special task >>>>> that needs to be executed only once during initialization: >>>>> > if a special `admin` account does not yet exist in the database, it >>>>> should be created >>>>> >>>>> Originally I figured that I could read from the MongoDB master server >>>>> on each node and check if the admin account already exists. If it does >>>>> not >>>>> then another node has not yet created it, so this node should do so. >>>>> However this is problematic because creating an admin password hash is >>>>> asynchronous and takes time. Therefore there is a delay between when a >>>>> node >>>>> decides to create the account and when the account is being found by >>>>> other >>>>> nodes when querying the database. >>>>> >>>>> The code snippet that reads from the Mongo master only and creates the >>>>> account is available here: >>>>> https://gist.github.com/**3839429<https://gist.github.com/3839429> >>>>> >>>>> In the future I would also like a special task to be executed every 5 >>>>> minutes. This task must then only be executed by a running server, and >>>>> not >>>>> by all servers. >>>>> >>>>> In short: when running a cluster of servers, how do you coordinate >>>>> between these servers which of them is going to execute a sole task such >>>>> as >>>>> the one described above? >>>>> >>>>> Tom >>>>> >>>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to nod...@googlegroups.com<javascript:> >>> To unsubscribe from this group, send email to >>> nodejs+un...@googlegroups.com <javascript:> >>> For more options, visit this group at >>> http://groups.google.com/group/nodejs?hl=en?hl=en >>> >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to nod...@googlegroups.com<javascript:> >> To unsubscribe from this group, send email to >> nodejs+un...@googlegroups.com <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> > > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
