I'd start here: http://doc.scrapy.org/en/latest/ You can read the code here: https://github.com/scrapy/scrapy
Other than that, there's a scrapy-dev google group ( https://groups.google.com/forum/#!forum/scrapy-developers ) that's more likely to answer this question correctly, but here's what I know - (I believe) Blocking at all would have to be programmed in - Scrapy is a single, async process - I don't know what the memory usage is, but I've run a dozen or so spiders on an old laptop without problem. -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/groups/opt_out.
