Hello,
If you're talking about changing headers of Request instances produced by
CrawlSpider rules,
you can use Rule's `process_request`
<http://doc.scrapy.org/en/latest/topics/spiders.html#crawling-rules>
argument to pass a method that does something on the request, for example
changing HTTP headers
That would be the recommended way.
The docs say that you should avoid overriding parse() method, because
that's where all the Rules' magic happen.
But you can still override it if you do it like this for example:
class MySpider(CrawlSpider):
name = 'example.com'
...
rules = (
...
)
def parse(self, response):
for r in super(MySpider, self).parse(response):
if isinstance(r, scrapy.Request):
# do something with the request...
yield r
Hope this helps,
Paul.
On Friday, March 4, 2016 at 10:59:44 AM UTC+1, 林子言 wrote:
>
> the doc said that cant override the CrawlSpider parse function,which
> leads to you cant pass headers to scrapy.Request
>
> how to solve this ?
>
> many thanks
>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
