I guess your spider might be banned by the website. Try this: http://doc.scrapy.org/en/latest/faq.html?highlight=banned#how-can-i-prevent-my-scrapy-bot-from-getting-banned
On Monday, June 9, 2014 4:04:11 AM UTC+8, Wei Lo wrote: > > I am now using scrapy to crawl massive data from a website. I encountered > the following problem: > > When I want to crawl data from XXX.com/a, the response code is 200 and it > works without any problem. > > Yet when I want to crawl data from XXX.com/b, the response code is ALWAYS > 403. Remember that XXX indicates the same domain. > > I use chrome browser to take a look at XXX.com/b, it works without any > problem. > > Can somebody give me comments on what is going on here? Thank you!!! > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
