Hi all, I'm using proxy to crawl a site, and it randomly gives me a bunch of error messages like:
2014-10-20 05:26:10-0800 [foo.bar] DEBUG: Retrying <GET http://foo.bar/foobar> (failed 1 times): [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>, <twisted.python.failure.Failure <class 'twisted.web.http._DataLoss'>>] I think this mostly happens with a bad proxy but it sometimes occurs with a healthy proxy as well. The thing is that this not only skips the url entry to crawl. I implemented a middleware(especially for proxy and retry middlewares), but it's really hard to catch this exception on scrapy level. Has anyone had the same issue? Thanks! -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
