FWIW, here you have a crochet+scrapy integration: https://github.com/rolando/scrapydo/blob/master/scrapydo/api.py#L90
Rolando On Tue, Jul 19, 2016 at 8:01 AM, <[email protected]> wrote: > Folks, I am having a tough time to integrate scrapy asynchronous > operations in django. > > I see little support for twisted, scrapy's underlying asynchronous > library, in the django world. > > I have tried to use the hendrix and crochet packages but found the > documentation and examples lacking. > > Naturally, I would expect to use celery with django, for asynchronous > tasks. It has an amazing integration. > > > My questions therefore are: > - Can I use celery to run an asynchronous spider crawl, given that > twisted is used underneath? Is this even the right way to go? > - If I do use scrapy X celery, will I have scaling issues? (say, if I > wanted to crawl 1000 pages) > > Has anyone got any experience here? > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
