Hi,
See this example
http://doc.scrapy.org/en/latest/topics/images.html#custom-images-pipeline-example
and overwrite get_media_requests method and set Referer header.
from scrapy.contrib.pipeline.images import ImagesPipelinefrom scrapy.exceptions
import DropItemfrom scrapy.http import Request
class MyImagesPipeline(ImagesPipeline):
def get_media_requests(self, item, info):
for image_url in item['image_urls']:
yield Request(image_url,
*headers=**{'Referer':'http://your_referer_url'}*)
Regards.
Le jeudi 12 juin 2014 08:57:49 UTC+1, R. Hsiao a écrit :
>
> Hi guys,
>
> I am a newbie to Scrapy, I found the ImagesPipeline seems will not set
> Referer automatic while downloading images,
>
> Anyone knowss how I can crawl those sites which have blocked No-Referer
> requests?
>
> Thanks
>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
