Pedro, It depends on usage as well as where you're running the process. Faster machines will obviously be able to run more processes.
If you're doing more processing on a page, you won't be able to run as many crawlers. Most of the time, I just use scrapy for the crawling, save to s3, and process later. So most of the CPU time is for the crawl, meaning my usage is minimal. I can run 20-30 per CPU without getting above 80% usage. On another job, I did a lot of processing on the page, running on an old server, and could only run 3-4 at a time. I'd suggest running the instances, monitoring your CPU usage with top, and adjusting as needed. On Wed, Sep 24, 2014 at 1:05 PM, Nicolás Alejandro Ramírez Quiros < [email protected]> wrote: > What about, DEFAULT ? xD > > That depends on your usage. > > El miércoles, 24 de septiembre de 2014 15:56:29 UTC-3, Pedro Henrique > escribió: > >> Which the ideal max_proc_per_cpu and max_proc value? >> > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
