Hi guy, I'm trying to scrape a site. Spider managed to login (I can download profile page etc). hovewer when spider tries to access a page with data it get response with status 302 and is redirected to the main page of the site.
When I'm doing this manually with browser everything is good and browser get 200 response and page with data. What I've done so far: - analysed request headers with Firebug and reproduced exactly within spider - analysed cookies. Some of them were missing in spider because are generated by JS, added manually. So cookies are also identical. - by deleting cookies one by one, found 1 cookie that is responsible for session login. When delete it via firebug I got the same server behaviour as spider faces - 302 and redirect to main page. Funny thing is that this cookie is received and send by spider perfectly, according to debug info - IP is the same (running from the same machine, spider under vagrant) NEED HELP/ADVICE: What do i miss? Requests are very similar and server response is different What could be the reason? Thanks in advance, Evgeny -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
