Re: [nodejs] Distributed workers recommendations

2013-12-26 Thread Angel Java Lopez
Just entered in my radar, today: https://github.com/OptimalBits/bull It uses redis for persistence. Each queue has name, redis server, redis port. And you can send any message to a named queue And process any message from a named queue, I suppose from any machine that can access redis server.

Re: [nodejs] Distributed workers recommendations

2013-12-26 Thread Daniel Rinehart
At my last company we had good success with coffee-resque backed by Redis with a little custom coding around it. Each worker was an independent process that was managed and kept alive with node-foreman and monit. Jobs added to the queue included a timeout that the worker used to trigger timeouts al

Re: [nodejs] Distributed workers recommendations

2013-12-26 Thread Angel Java Lopez
Sorry, I don't know how to express in more detail. But the async is at FE server, something like sendtask(task, function (err, result) { ... .}); The task (maybe a simple javascript object, serializable to JSON), goes to the task queue, and the callback is assigned to task id. If the response q

Re: [nodejs] Distributed workers recommendations

2013-12-26 Thread AlainM
How does this fit in NODE.JS?? What you describe does, not seem like async programming at all to me... Angel Java Lopez escreveu: >I prefer > >Available worker pulls task > >over > >Available worker is notified > >The first one could use a queue, as you mentioned. So any other process >(a >prog

Re: [nodejs] Distributed workers recommendations

2013-12-26 Thread Angel Java Lopez
I prefer Available worker pulls task over Available worker is notified The first one could use a queue, as you mentioned. So any other process (a programmer, a supervisor program, etc...) can launch workers, and the workers only need to know: - The queue having pending tasks - Maybe a queue wh

[nodejs] Distributed workers recommendations

2013-12-25 Thread Floby
This is an interesting subject. I would look into the ømq guide for ideas, especially the part on pipelines. Even if you don't chose ømq as your supporting technology, their docs have some very useful material for distributed architectures. -- -- Job Board: http://jobs.nodejs.org/ Posting gui

[nodejs] Distributed workers recommendations

2013-12-25 Thread Tim Shnaider
Hi Node.js experts, There are many many modules and frameworks able to solve this problem. I'm looking for CONCRETE recommendations and guidance for well supported active implementations. e.g. pubsub, message queues etc I'm a bit overwhelmed by the options and amount of research testing I'd h