Hello everyone, looking for any suggestions/ideas on this matter : I am running scrapyd 1.1 + scrapy 0.24.6 with a single spider that crawls over one domain. The development machine that host scrapyd's instance has 4 cores processor. I would like a setup to process N jobs simultaneously but scrapyd is processing 1 to 4 at a time regardless of how many jobs are pending and the max_proc_per_cpu parameter. Also i have noticed that scrapyd indeed use the 4 cores while working on 4 process at a time but would like to improve performance for batch execution using most of the processor capacity as possible and currently is using and maximum of 26% of each core.
More info: http://stackoverflow.com/questions/30672910/parallelism-performance-problems-with-scrapyd-and-single-spider -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
