I have quite a few sites that I'm trying to crawl that follow this pattern: The scrape begins, within seconds Scrapy reports "finished successfully", but the results JSON file is empty. I added some logging to dupefilter.py just to see if anything is crawled, and nothing is. I have "respect_nofollow" set to true, and am on Scrapy version 0.24.4. Any thoughts would be greatly appreciated. Thanks.
-- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
