You should be able to use queuing libraries. I know ZeroMQ works fine with Node. Here are some resources on this: * http://stackoverflow.com/questions/4700935/what-are-good-message-queue-options-for-nodejs * http://www.quora.com/Node-js/What-the-best-way-to-have-delayed-job-queue-with-node-js
I would generally stay away from rolling my own unless I'm forced to. Moving on, how do you wish to expose an end-point? Restful API or Socket event? Something else? The way you describe it, you have multiple tasks broken into smaller jobs. Typically I'd like to use something like GridGain for such a requirement, especially if it is resource intensive. Mind you GridGain is JVM based and which performs better is a big debate in itself. :) You can try Amazon SQS, I don't have any experience but it should be a fairly safe option. If it won't constraint you, I'd say go for it. - Sri Srirangan | +91 9711 477 595 | About <http://srirangan.net/about> GitHub <https://github.com/Srirangan> LinkedIn<http://www.linkedin.com/in/srirangan> Twitter <http://twitter.com/srirangan> | Review19 <http://review19.com> "Collaborate & Track Decisions" On Sat, Jul 28, 2012 at 2:20 AM, Baz <[email protected]> wrote: > Hello, > > I have a process where a 3rd party service sends a message to a > pre-defined endpoint in my system. This triggers a series of actions on my > side: > > 1. Invoke a webservice to pull a large batch of items that need to be > processed > 2. Iterate through the large batch and queue up a number of smaller > batches to process > 3. Process each smaller batch in semi-parallel: > 1. Run calculations > 2. Save result to db > 3. Dynamically update any web clients "watching" the affected data > when complete > > It seems there are two discrete parts here: that which listens for, and > processes messages - and a second part that dynamically renders changes to > the client based on changes in the backend. > > I would love some advice, thoughts or suggestions on how to architect any > of this, especially the first part regarding listening for and processing > messages. Some basic thoughts: > > I need an endpoint, that's no problem, but then it seems I need some sort > of queuing system. Should I roll my own? What about Amazon SQS? Is it easy > to get node to respond to messages and spawn more instances if need be? > What are some important considerations? > > For the front-end, should I be basically looking at Socket.io? > > Thanks for any help, > Baz > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
