Interesting. I wonder if it is better to use phantomjs for the PNG rendering. Although it is slower, I don't need child_process
On Fri, May 20, 2016 at 5:42 PM Zlatko <[email protected]> wrote: > You have multiple things there. Thousands requests (per second, I assume) > is a lot even without spawning a child shell. Thousands of spawned shells, > even without image magick, can bring a busy system down. But then again, > you will know best with a test. Try it and see if and where you get stuck. > > Even if the system itself handles it, depending on how fast you get to > your saturation limits, your response time may get increasingly longer. If > that's acceptable and you know there will be downtimes when the jobs can > catch up, you might do well with this. > > The bigger problem I see there is that you cannot scale horizontally > there. You wanna be able to send those convert jobs to other machines as > well, probably. And for that, a simple child_process won't do, it would be > better if you had some sort of a job queue (MQ, redis pubsub, something of > the sort depending if you can allow failures and lost jobs). Then your > service that listens for reqs only shoots them to the queue, and listens > for a matching response on the response queue. And your actual png > generation service can listen on the other side and only take as many > requests simultaneously as it makes sense - e.g. maybe per num of CPUs or > similar, test there and see what makes sense. And when you have a lot of > work, just spawn another png generator on a new machine and you're good, no > blocking. > > The best thing to do is simply measure. Try to run several processes at > once and shoot a thousand requests, see how long it takes for all of them > to finish. Thn try to fire 10 waves of 1k reqs, each wave 2 seconds apart, > and see how that goes. (Of course, adjust the numbers to your environment.). > > Lastly, from my personal experience, these kinds of things regularly grow > out of hand and original specs, and it's better to just do job queues from > the get-go. > > Hope that helps. > > Zlatko > > > On Thursday, May 19, 2016 at 11:51:46 PM UTC+2, Alisson Cavalcante Agiani > wrote: >> >> I need to develop a service where the user sends an array of values, I >> generate a chart with d3.js and convert it to png with a child_process >> calling imagemagick to send it back to the user. Are there any caveats for >> child_process when scaling for thousands of requests? >> > -- > Job board: http://jobs.nodejs.org/ > New group rules: > https://gist.github.com/othiym23/9886289#file-moderation-policy-md > Old group rules: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/nodejs/09ff5e6e-f2fe-45d7-bb57-488230ee78d0%40googlegroups.com > <https://groups.google.com/d/msgid/nodejs/09ff5e6e-f2fe-45d7-bb57-488230ee78d0%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CACZE8Y6cqWvVTFF8Ee9SdvsBZTk-DKraTACCs_hq%2BWciTGbRcA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
