We use redis for this, specifically the coffee-resque module. https://github.com/technoweenie/coffee-resque
Jobs get put on the queue and we have workers pick them up to be processed. You could then use something like socket.io to send messages out to clients that whatever you processed has been processed. Some of the other suggestions here would also work fine. We were already using redis so we just piggybacked off of that. - Tim On Friday, July 27, 2012 4:50:55 PM UTC-4, Baz wrote: > > Hello, > > I have a process where a 3rd party service sends a message to a > pre-defined endpoint in my system. This triggers a series of actions on my > side: > > 1. Invoke a webservice to pull a large batch of items that need to be > processed > 2. Iterate through the large batch and queue up a number of smaller > batches to process > 3. Process each smaller batch in semi-parallel: > 1. Run calculations > 2. Save result to db > 3. Dynamically update any web clients "watching" the affected data > when complete > > It seems there are two discrete parts here: that which listens for, and > processes messages - and a second part that dynamically renders changes to > the client based on changes in the backend. > > I would love some advice, thoughts or suggestions on how to architect any > of this, especially the first part regarding listening for and processing > messages. Some basic thoughts: > > I need an endpoint, that's no problem, but then it seems I need some sort > of queuing system. Should I roll my own? What about Amazon SQS? Is it easy > to get node to respond to messages and spawn more instances if need be? > What are some important considerations? > > For the front-end, should I be basically looking at Socket.io? > > Thanks for any help, > Baz > > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
