> [Disclaimer: My third client implementation makes me a serial customer of theirs. I would probably stick a Transloadit sticker on my laptop so I guess that makes me a fanboy]
Thanks : ). Glad to hear our service is working well for you! For those determined to engineer their own solution, here is some advise regardless: Using `nice` will help in the beginning, but if you get serious load, you'll end up with a *lot* of processes, and while they may not be able to eat up your CPU, they will eat up your memory, and the OOM Killer will decide who has to pay for it. Before that however you're likely to swap which will bring all your disk I/O to a crawl. Using nice in addition with a queue / similar however can be ok. Monitor your child processes: We've seen image magick, ffmpeg and various other command line utilities entering states where they'll start eating either memory or CPU time like crazy without completing significant or sometimes any amount of work. You need to be able to detect this and kill the process you spawned. If you use a node addon / binding, you may not be able to do that unless you do the work in another process, so IMO exec/spawn are your friends. Use a queue: I'm partial to redis-based queue implementations like https://github.com/technoweenie/coffee-resque or http://learnboost.github.com/kue/, but anything that will allow you to asynchronously process varying workloads with minimal overhead is fine in my book. Use Amazon S3 as a temporary file system: When you get a new file, put it there (use reduced redundancy if you want to save costs), then let the workers pull the file from S3. You will thank yourself for doing this when it comes to scaling (especially down), as you don't have to keep a reference count for files on your various upload machines. Use your tools well: You'd be surprised by the performance / quality gains you can get from image magick, ffmpeg and friends by using the right options, so it's definitely worth to do a bit of research in that direction for your particular needs. As mentioned by Richard: We run a SaaS/IaaS product for these kind of problems over at transloadit.com. It may not fit all use cases, but you can definitely save a lot of development time if your use case is covered. --fg PS: I still have nightmares from the last time I used CentOS, but YMMV : ). -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
