Hi Samir,
I solved that problem for CPU intensive jobs. I ended up with my own
control flow library, because no one has been gone in the route of
"queuing" task with a work-in-progress limit in a sane way before:
https://github.com/mcollina/kanban. There is async.queue, but it lacks some
flexibility, if you have a processing pipeline with different constraint.
The number of worker processes you want to spawn in a given period of time
is strictly dependent to the number of concurrent process you might want to
run in your system. With node.js you can end up prettly quickly with load >
1, and then your performance gets worse as more processes are added.
Let me know if you need more help on this subject.
Cheers,
Matteo
2013/3/25 Samir Sabri <as3...@gmail.com>
> Thanks Floby, I appreciate your advice, do you know a good blog or source
> that talks about workers and jobs queuing?
>
> On Mon, Mar 25, 2013 at 10:52 AM, Floby <florent.j...@gmail.com> wrote:
>
>> no. Definitely not.
>> Spawn a predefined number of workers (using the cluster api) and queue
>> jobs in the master process then send them to the workers when they are
>> ready to process.
>>
>>
>> On Monday, 25 March 2013 06:49:32 UTC+1, simo wrote:
>>
>>> So, is it a recommended approach to spawn a new process for each opened
>>> socket?
>>>
>>> On Sun, Mar 24, 2013 at 11:24 PM, Alan Hoffmeister <alanhof...@gmail.com
>>> > wrote:
>>>
>>>> I think that the maximum number of opened files is the only unix's
>>>> limitation for the number of sockets, and this can be changed.
>>>>
>>>> --
>>>> Att,
>>>> Alan Hoffmeister
>>>>
>>>>
>>>> 2013/3/24 simo <as3...@gmail.com>
>>>>
>>>>> I have to serve a calculation via algorithm, I've been advised to
>>>>> use a child process per each opened socket, what I am about to do is
>>>>> something like that:
>>>>>
>>>>> var spawn = require('child_process').spawn**;var child = spawn('node',
>>>>> ['algorithem.js']);
>>>>>
>>>>> I know how to send argument to the algorithm process and how to
>>>>> receive results.
>>>>>
>>>>> What I am concerned about, is how many socket (each socket will spawn
>>>>> a process) I can have? How can I resolve this with my cloud hosting
>>>>> provider? so that my app gets auto scaled?
>>>>>
>>>>> What's the recommended node js cloud hosting provider?
>>>>>
>>>>> Finally, is this a good approach in using child processes?
>>>>>
>>>>> --
>>>>> --
>>>>> Job Board: http://jobs.nodejs.org/
>>>>> Posting guidelines: https://github.com/joyent/**
>>>>> node/wiki/Mailing-List-**Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines>
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "nodejs" group.
>>>>> To post to this group, send email to nod...@googlegroups.com
>>>>>
>>>>> To unsubscribe from this group, send email to
>>>>> nodejs+un...@**googlegroups.com
>>>>>
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en>
>>>>>
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "nodejs" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to nodejs+un...@**googlegroups.com.
>>>>>
>>>>> For more options, visit
>>>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>>>>> .
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> --
>>>> Job Board: http://jobs.nodejs.org/
>>>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List-
>>>> **Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines>
>>>> You received this message because you are subscribed to the Google
>>>> Groups "nodejs" group.
>>>> To post to this group, send email to nod...@googlegroups.com
>>>>
>>>> To unsubscribe from this group, send email to
>>>> nodejs+un...@**googlegroups.com
>>>>
>>>> For more options, visit this group at
>>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en>
>>>>
>>>> ---
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "nodejs" group.
>>>> To unsubscribe from this topic, visit https://groups.google.com/d/**
>>>> topic/nodejs/qP_hsLZMq1A/**unsubscribe?hl=en<https://groups.google.com/d/topic/nodejs/qP_hsLZMq1A/unsubscribe?hl=en>
>>>> .
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> nodejs+un...@**googlegroups.com.
>>>>
>>>> For more options, visit
>>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>>>> .
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --
>>> Kind Regards,
>>> ------------------------------**---------------
>>> Samir Sabri
>>> Software Architect& Developer
>>> www.dcaclab.com
>>> Jordan-Middle East
>>>
>> --
>> --
>> Job Board: http://jobs.nodejs.org/
>> Posting guidelines:
>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> You received this message because you are subscribed to the Google
>> Groups "nodejs" group.
>> To post to this group, send email to nodejs@googlegroups.com
>> To unsubscribe from this group, send email to
>> nodejs+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "nodejs" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/nodejs/qP_hsLZMq1A/unsubscribe?hl=en.
>> To unsubscribe from this group and all its topics, send an email to
>> nodejs+unsubscr...@googlegroups.com.
>>
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>
>
> --
> --
> Kind Regards,
> ---------------------------------------------
> Samir Sabri
> Software Architect& Developer
> www.dcaclab.com
> Jordan-Middle East
>
> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nodejs@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nodejs+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
