Hi Leonardo, Scrapy doesn't execute parse callbacks in parallel, only a single parse callback can be executed at time.
понедельник, 15 июня 2015 г., 12:25:25 UTC+5 пользователь Leonardo Casanova написал: > > I have also asked this question in stackoverflow. > > > http://stackoverflow.com/questions/30808294/middleware-to-limit-the-amount-of-requests-downloaded-by-scrapy > > On Friday, June 12, 2015 at 6:42:49 PM UTC+2, Leonardo Casanova wrote: >> >> Hi, >> >> As the title says my problem is that I need to limit the amount of parse >> callbacks being processed concurrently as the are quite memory intensive >> (some of them create a selenium instance). >> To avoid this I want to prevent scrapy from downloading new requests >> while parse callbacks are still being processed. >> >> I have written this middleware to reschedule requests if they exceed a >> certain limit. >> However at some point the scraping seems to stop and no new requests are >> made. >> Can you help me check if I missed something? >> >> Best Regards >> Leo >> > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
