Mike, I'd look for the pub-sub setup that makes the most sense to you, and keep it simple. You don't need a ton of "framework" on top of it. Redis will meet your needs well, but if you need many thousands of messages a second (which it doesn't sound like is the case!), I would take a closer look at ZeroMQ.
The other nice aspect of Redis? It's already a badass and simple key-value store, so it can be used in plenty of other useful ways without adding more complexity to the operations of your application. On Mon, Feb 18, 2013 at 5:57 PM, Jeremy Darling <[email protected]> wrote: > Depending on your constraints my pet project MongoMQ > (https://github.com/jdarling/MongoMQ/) might do what your looking for. Its > a simple MQ built using MongoDB and tailed cursors. Has proven to handle > several 1000 messages a second without blowing up, and is in use at my work > for quite some time now. > > You might give it a look. It was basically inspired by Hook.io to some > extent, but built to be as thin as possible and still be very flexible. Any > questions feel free to ask. > > - Jeremy > > > On Mon, Feb 18, 2013 at 7:18 PM, MikeB_2012 <[email protected]> wrote: >> >> Hi Alexey. Good questions! Answers: >> >> a) short term: me (small number of processes). Baby steps. Long term: >> there needs to be an awareness in each process of at least a subset of the >> others and something that cues rebirth. Rebirth to an initial configuration >> would be the duty of another process; >> b) which data? >> c) this is part of my learning. In the physical world how is the data >> represented? When I fake the system it is the equivalent of all processes >> broadcasting 'to all' or all information being posted to a common >> 'blackboard'. I thought maybe I could do the same, say using CloudDB (which >> I'm familiar with) or an equivalent. But it just seemed to me that this >> would make the DB a bottleneck as it handles information pushed to it and >> pulled from it. But maybe that's okay? >> d) see a) >> e) # of processes is task dependent and self-regulating as the system >> receives feedback on its performance. Questions of load ... I'm still >> learning so I've no idea. I'm still very ignorant of real world of the >> hardware considerations for distributed real-world processing. >> >> You're correct, I can fake it (and have been to a certain degree). The >> problem is that faking it leads to a predictable, brittle system. Also, a >> faked system has no asynchrony and has complete information. However, as >> you may have guessed, the functionality I'm seeking is to yield a dynamic, >> slightly unpredictable, but potentially adaptive system. So far, I've found >> that faking the system is hard and part of the reason I've looked to a real >> construct; probably I'll just be trading one set of challenges for a >> different set, but I'll be learning something in the process so it's all >> good :-) >> >> >>> >>> There are questions like >>> >>> - who will keep an eye on monitoring and restarting died agents. >>> - where to store data? >>> - what to do if data sent to died agent? Should it be recovered and >>> restarted? >>> - how to restart and restore state of died agents? >>> - how much agents start on one machine? how to measure, distribute and >>> balance load? How to add new machine or fix broken? >>> ... >>> >>> And all this becomes even more complex if there are requirements for >>> transactions or atomicity of operations. >>> >>> I'd suggest to not to built such system and instead fake it with simpler >>> approaches. I.e. message queues, databases, stateless workers. >> >> -- >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "nodejs" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
