You can write a downloader middleware and catch them in its
process_exception()
On 15-12-4 下午9:15, 洪翔 wrote:
|
Useproxy,andfinalscrapyd log
stats:2015-12-0421:03:46[scrapy]INFO:DumpingScrapystats:{'downloader/exception_count':332,'downloader/exception_type_count/scrapy.core.downloader.handlers.http11.TunnelError':21,'downloader/exception_type_count/twisted.internet.error.TimeoutError':229,'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived':82,...}
|
|
def create_foo_request():
|
|
return Request(url=url, callback=callback, headers=headers,
meta=meta_merged,
|
|
errback=self._request_errback)
|
|
def _request_errback(self, failure):
|
|
# Expect to handle TunnelError, TimeoutError, ResponseNeverReceived
|
|
But can't catch any error at _request_errback(...)
|
--
You received this message because you are subscribed to the Google
Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
