SOLVED: I added to form data all the inputs in the page and it worked. On Saturday, February 22, 2014 9:23:09 PM UTC-7, Eli Mor wrote: > > I'm new to this, and tried to find help in other posts. The post from Duy > Nguyen https://groups.google.com/d/msg/scrapy-users/aVlsOz7972I Was > pretty close, but i'm still having issues getting the information I want. > I'm trying to scrape results from award miles for the united website at > http://www.united.com/web/en-US/apps/booking/flight/searchAward.aspx but > the results are being redirected and the following code is not able to get > the correct response. > > from scrapy.spider import Spider > from scrapy.selector import Selector > from scrapy.http import FormRequest, Request > > class DmozSpider(Spider): > name = "united" > allowed_domains = ["united.com"] > start_urls = [ > "http://www.united.com/"; > ] > > def parse(self, response): > return [FormRequest(url=" > http://www.united.com/web/en-US/apps/booking/flight/searchAward.aspx";, > method='POST', > formdata={ > > 'ctl00$ContentInfo$SearchForm$Airports1$Destination$txtDestination':'New > York/Newark, NJ (EWR - Liberty)', > > 'ctl00$ContentInfo$SearchForm$Airports1$Origin$txtOrigin':'San > Francisco, CA (SFO)', > > 'ctl00$ContentInfo$SearchForm$DateTimeCabin1$Depdate$txtDptDate': > '8/19/2014', > > 'ctl00$ContentInfo$SearchForm$DateTimeCabin1$Depdate1$txtDptDate': > '8/27/2014', > 'ctl00$ContentInfo$SearchForm$searchBy$SearchBy': > 'rdosearchby3', > }, > #dont_click=True, > callback=self.parse_award)] > > def parse_award(self, response): > print "status: %s" % response.status > print "headers: %s" % response.headers > print "redirect_urls: %s" % response.meta > filename = response.url.split("/")[-2] > open('resp.html', 'wb').write(response.body) > > The response to the post request says the status is 200 ('OK'), but when > using firebug for the same query, it shows the post status as 302 ('Moved > Temporarily'), and the results being available in > http://www.united.com/web/en-US/apps/booking/flight/searchResultAward1.aspx > > Any help pointing in the right direction is appreciated! >
-- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/groups/opt_out.
