Here is your answer:
https://github.com/scrapy/scrapy/blob/master/scrapy/contrib/closespider.py#L60
El lunes, 3 de noviembre de 2014 02:50:20 UTC-2, Sungmin Lee escribió:
>
> Hi all,
>
> I've built my own middleware, and I would like to stop crawler when it
> meets some condition.
> I know that from spider, I can terminate the process by raising
> CloseSpider() exception, but this does not work from middleware's
> process_request() function at all.
>
> I googled this topic for a while, but none of the solutions I found worked
> so far.
> (raise CloseSpider('message'), spider.close_down = True, etc.)
>
> Even sys.exit(0) call doesn't work shouting exception tracebacks:
>
> Traceback (most recent call last):
> File
> "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/core/engine.py",
>
> line 137, in _next_request_from_scheduler
> d = self._download(request, spider)
> File
> "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/core/engine.py",
>
> line 213, in _download
> dwld = self.downloader.fetch(request, spider)
> File
> "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/core/downloader/__init__.py",
>
> line 87, in fetch
> dfd = self.middleware.download(self._enqueue_request, request, spider)
> File
> "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py",
>
> line 65, in download
> deferred = mustbe_deferred(process_request, request)
> --- <exception caught here> ---
> File
> "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/utils/defer.py",
>
> line 39, in mustbe_deferred
> result = f(*args, **kw)
> File
> "/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py",
>
> line 32, in process_request
> response = method(request=request, spider=spider)
> File "/Users/username/Projects/projectname/mymiddleware.py", line 56, in
> process_request
> sys.exit(0)
> exceptions.SystemExit: 0
>
> Is there any way that I can elegantly finish the scraper from
> process_request function in custom middleware?
> Thanks!
>
>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
