So, scrapy just crawls when you debug? Maybe not related, but it could be your problem, Scrapy redirect automatically and executes `parse` method only on the final page, not on the ones that contains the redirects. Moreover scrapy doesn't crawl duplicated URLs and it doesn't log anything but the first time.
El jueves, 18 de junio de 2015, 16:04:47 (UTC-5), outoftheblue9 escribió: > > I am using the RedirectMiddleware and the redirected pages are not getting > crawled. If I put a break point in the middlewares class and do some > steping, then let it go, it crawls to the redirected URL. Anyone have any > issues like this? > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
