I am now using scrapy to crawl massive data from a website. I encountered the following problem:
When I want to crawl data from XXX.com/a, the response code is 200 and it works without any problem. Yet when I want to crawl data from XXX.com/b, the response code is ALWAYS 403. Remember that XXX indicates the same domain. I use chrome browser to take a look at XXX.com/b, it works without any problem. Can somebody give me comments on what is going on here? Thank you!!! -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
