Sorry, I should have been more clear: Mongo in a cluster (as Dan points out) is certainly not `always consistent`. I got the impression that your implementation had many node servers reading from a single mongo server. If thats the case, then that one server will be consistent with itself :D
Sorry for the confusion! On Friday, October 5, 2012 at 4:14 PM, Dan Milon wrote: > Mongo is definitely NOT always consistent. > > On Fri, Oct 5, 2012 at 10:14 PM, Evan <evantah...@gmail.com > (mailto:evantah...@gmail.com)> wrote: > > You are basically proposing the schema for DelayedJobs (Ruby) [ > > https://github.com/collectiveidea/delayed_job ]. This kind of thing gets > > really weird in eventually consistant databases, but Mongo (like mySQL) is > > always consistant so you should be OK. However, a lot of folks have been > > finding some locking problems with this approach (multiple job execution or > > really aggressive table locking is needed), so most folks tend to use a > > store which can support atomic "push" and "pop" operations so you can > > ensure that one and only one worker gets the job. Redis is the most > > popular of these types of stores these day. I make use of that property in > > http://actionherojs.com/ for exactly this purpose, as does the very popular > > https://github.com/defunkt/resque and some other projects. > > > > > > On Friday, October 5, 2012 12:32:39 PM UTC-7, Mark Hahn wrote: > > > I may be crazy but I'm implementing a scheme where processes get the > > > tasks from a db record and then stores their process number in that > > > record. Then while they are running they periodically check to make sure > > > that their server number is still the one in the record. If another > > > process has 'stolen' the task then the process aborts and looks for > > > another one to do. > > > > > > This is the only way I could figure out how to do task assignment when > > > faced with a db that only has "eventual consistency". The CouchDB i'm > > > using offers no atomic operations so this was the only reliable way to do > > > it. > > > > > > It works quite well. In the usual case it just grabs the task, does it, > > > and moves on. Collisions are rare, but they may be more frequent as the > > > cluster grows in size. > > > > > > > > > On Fri, Oct 5, 2012 at 9:21 AM, Dan Milon <danm...@gmail.com> wrote: > > > > greelkorke ment using a job queue where jobs are put, and handed to > > > > workers. > > > > > > > > If you want to do it only with mongo, you'll need to use some "lock" > > > > document, that is set and unset by the first process which tries to > > > > initiate a task. All other processes which try to grab the lock while > > > > its held by another process should assume that the job is being worked > > > > by another process. But thats really ugly & has problems because jobs > > > > cant be acknowledged, so if a process crashes while its performing some > > > > task, you're fucked. > > > > > > > > On Fri, Oct 5, 2012 at 5:07 PM, Tom <tomm...@gmail.com> wrote: > > > > > Unfortunately I'm afraid that I don't see how a scheduler can avoid > > > > > the concurrency problems. Note that the advantages (e.g. in > > > > > availability) of having a cluster should be maintained here, and so > > > > > you cannot run a scheduler in a separate process on a single server. > > > > > If every server would be running the scheduler, the same concurrency > > > > > problems would arise. What were you proposing? > > > > > > > > > > About the initialization, I guess that would work. It is not the way > > > > > I would prefer to do it, as I would like the application to be > > > > > self-controlled and usable without running special tools, but I > > > > > reckon it is an acceptable approach. > > > > > > > > > > Tom > > > > > > > > > > Op vrijdag 5 oktober 2012 19:59:33 UTC+7 schreef greelgorke het > > > > > volgende: > > > > > > i wouldn't do it that way. when deploying your app just do a > > > > > > pre-start script, that ensures the existence of your desired data. > > > > > > If you have periodical tasks, it's best to use a lib for it, that > > > > > > triggers jobs appart of your main application. i.E > > > > > > http://stackoverflow.com/questions/3785736/is-there-a-job-scheduler-library-for-node-js > > > > > > , so you just avoid the concurrency problems. > > > > > > > > > > > > Am Freitag, 5. Oktober 2012 14:04:58 UTC+2 schrieb Tom: > > > > > > > I've setup a cluster of physical servers. Each server runs > > > > > > > exactly the same code. Moreover, each server runs multiple node > > > > > > > processes using the build in cluster functionality. > > > > > > > > > > > > > > I use MongoDB (native) to share information between processes and > > > > > > > servers. However, I am having some difficulty with running a > > > > > > > special task that needs to be executed only once during > > > > > > > initialization: > > > > > > > > if a special `admin` account does not yet exist in the > > > > > > > > database, it should be created > > > > > > > > > > > > > > Originally I figured that I could read from the MongoDB master > > > > > > > server on each node and check if the admin account already > > > > > > > exists. If it does not then another node has not yet created it, > > > > > > > so this node should do so. However this is problematic because > > > > > > > creating an admin password hash is asynchronous and takes time. > > > > > > > Therefore there is a delay between when a node decides to create > > > > > > > the account and when the account is being found by other nodes > > > > > > > when querying the database. > > > > > > > > > > > > > > The code snippet that reads from the Mongo master only and > > > > > > > creates the account is available here: > > > > > > > https://gist.github.com/3839429 > > > > > > > > > > > > > > In the future I would also like a special task to be executed > > > > > > > every 5 minutes. This task must then only be executed by a > > > > > > > running server, and not by all servers. > > > > > > > > > > > > > > In short: when running a cluster of servers, how do you > > > > > > > coordinate between these servers which of them is going to > > > > > > > execute a sole task such as the one described above? > > > > > > > > > > > > > > Tom > > > > > > > > > > -- > > > > > Job Board: http://jobs.nodejs.org/ > > > > > Posting guidelines: > > > > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > > > > You received this message because you are subscribed to the Google > > > > > Groups "nodejs" group. > > > > > To post to this group, send email to nod...@googlegroups.com > > > > > > > > > > To unsubscribe from this group, send email to > > > > > nodejs+un...@googlegroups.com > > > > > > > > > > For more options, visit this group at > > > > > http://groups.google.com/group/nodejs?hl=en?hl=en > > > > > > > > -- > > > > Job Board: http://jobs.nodejs.org/ > > > > Posting guidelines: > > > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > > > You received this message because you are subscribed to the Google > > > > Groups "nodejs" group. > > > > To post to this group, send email to nod...@googlegroups.com > > > > > > > > To unsubscribe from this group, send email to > > > > nodejs+un...@googlegroups.com > > > > > > > > For more options, visit this group at > > > > http://groups.google.com/group/nodejs?hl=en?hl=en > > > > > -- > > Job Board: http://jobs.nodejs.org/ > > Posting guidelines: > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > You received this message because you are subscribed to the Google > > Groups "nodejs" group. > > To post to this group, send email to nodejs@googlegroups.com > > (mailto:nodejs@googlegroups.com) > > To unsubscribe from this group, send email to > > nodejs+unsubscr...@googlegroups.com > > (mailto:nodejs%2bunsubscr...@googlegroups.com) > > For more options, visit this group at > > http://groups.google.com/group/nodejs?hl=en?hl=en > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > (mailto:nodejs@googlegroups.com) > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > (mailto:nodejs+unsubscr...@googlegroups.com) > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
