Mongo is definitely NOT always consistent. On Fri, Oct 5, 2012 at 10:14 PM, Evan <evantah...@gmail.com> wrote:
> You are basically proposing the schema for DelayedJobs (Ruby) [ > https://github.com/collectiveidea/delayed_job ]. This kind of thing gets > really weird in eventually consistant databases, but Mongo (like mySQL) is > always consistant so you should be OK. However, a lot of folks have been > finding some locking problems with this approach (multiple job execution or > really aggressive table locking is needed), so most folks tend to use a > store which can support atomic "push" and "pop" operations so you can > ensure that one and only one worker gets the job. Redis is the most > popular of these types of stores these day. I make use of that property in > http://actionherojs.com/ for exactly this purpose, as does the very > popular https://github.com/defunkt/resque and some other projects. > > > On Friday, October 5, 2012 12:32:39 PM UTC-7, Mark Hahn wrote: > >> I may be crazy but I'm implementing a scheme where processes get the >> tasks from a db record and then stores their process number in that record. >> Then while they are running they periodically check to make sure that >> their server number is still the one in the record. If another process has >> 'stolen' the task then the process aborts and looks for another one to do. >> >> This is the only way I could figure out how to do task assignment when >> faced with a db that only has "eventual consistency". The CouchDB i'm >> using offers no atomic operations so this was the only reliable way to do >> it. >> >> It works quite well. In the usual case it just grabs the task, does it, >> and moves on. Collisions are rare, but they may be more frequent as the >> cluster grows in size. >> >> >> On Fri, Oct 5, 2012 at 9:21 AM, Dan Milon <danm...@gmail.com> wrote: >> >>> greelkorke ment using a job queue where jobs are put, and handed to >>> workers. >>> >>> If you want to do it only with mongo, you'll need to use some "lock" >>> document, that is set and unset by the first process which tries to >>> initiate a task. All other processes which try to grab the lock while its >>> held by another process should assume that the job is being worked by >>> another process. But thats really ugly & has problems because jobs cant be >>> acknowledged, so if a process crashes while its performing some task, >>> you're fucked. >>> >>> On Fri, Oct 5, 2012 at 5:07 PM, Tom <tomm...@gmail.com> wrote: >>> >>>> Unfortunately I'm afraid that I don't see how a scheduler can avoid the >>>> concurrency problems. Note that the advantages (e.g. in availability) of >>>> having a cluster should be maintained here, and so you cannot run a >>>> scheduler in a separate process on a single server. If every server would >>>> be running the scheduler, the same concurrency problems would arise. What >>>> were you proposing? >>>> >>>> About the initialization, I guess that would work. It is not the way I >>>> would prefer to do it, as I would like the application to be >>>> self-controlled and usable without running special tools, but I reckon it >>>> is an acceptable approach. >>>> >>>> Tom >>>> >>>> Op vrijdag 5 oktober 2012 19:59:33 UTC+7 schreef greelgorke het >>>> volgende: >>>> >>>>> i wouldn't do it that way. when deploying your app just do a pre-start >>>>> script, that ensures the existence of your desired data. >>>>> If you have periodical tasks, it's best to use a lib for it, that >>>>> triggers jobs appart of your main application. i.E >>>>> http://stackoverflow.com/**q**uestions/3785736/is-there-a-**jo** >>>>> b-scheduler-library-for-**node-**js<http://stackoverflow.com/questions/3785736/is-there-a-job-scheduler-library-for-node-js>, >>>>> so you just avoid the concurrency problems. >>>>> >>>>> Am Freitag, 5. Oktober 2012 14:04:58 UTC+2 schrieb Tom: >>>>>> >>>>>> I've setup a cluster of physical servers. Each server runs exactly >>>>>> the same code. Moreover, each server runs multiple node processes using >>>>>> the build in cluster functionality. >>>>>> >>>>>> I use MongoDB (native) to share information between processes and >>>>>> servers. However, I am having some difficulty with running a special task >>>>>> that needs to be executed only once during initialization: >>>>>> > if a special `admin` account does not yet exist in the database, it >>>>>> should be created >>>>>> >>>>>> Originally I figured that I could read from the MongoDB master server >>>>>> on each node and check if the admin account already exists. If it does >>>>>> not >>>>>> then another node has not yet created it, so this node should do so. >>>>>> However this is problematic because creating an admin password hash is >>>>>> asynchronous and takes time. Therefore there is a delay between when a >>>>>> node >>>>>> decides to create the account and when the account is being found by >>>>>> other >>>>>> nodes when querying the database. >>>>>> >>>>>> The code snippet that reads from the Mongo master only and creates >>>>>> the account is available here: >>>>>> https://gist.github.com/****3839429<https://gist.github.com/3839429> >>>>>> >>>>>> In the future I would also like a special task to be executed every 5 >>>>>> minutes. This task must then only be executed by a running server, and >>>>>> not >>>>>> by all servers. >>>>>> >>>>>> In short: when running a cluster of servers, how do you coordinate >>>>>> between these servers which of them is going to execute a sole task such >>>>>> as >>>>>> the one described above? >>>>>> >>>>>> Tom >>>>>> >>>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List- >>>> **Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to nod...@googlegroups.com >>>> >>>> To unsubscribe from this group, send email to >>>> nodejs+un...@**googlegroups.com >>>> >>>> For more options, visit this group at >>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>> >>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List-* >>> *Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to nod...@googlegroups.com >>> >>> To unsubscribe from this group, send email to >>> nodejs+un...@**googlegroups.com >>> >>> For more options, visit this group at >>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>> >> >> -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
