Google use JavaScript to load results and because Scrapy itself doesn't support JavaScript, you can't see results.
Use this url: https://www.google.com/search?output=search&q=jquery Umair On Sunday, December 22, 2013 1:11:44 PM UTC+5, Gaurang shah wrote: > > Hi Guys, > > I have just started with the scrapy and facing a problem. Let me first > tell you what i am trying to develope. > > - search for the keyword on the google. - > > https://www.google.com/?q=keyword/#q=<https://www.google.com/?q=selenium/#q=selenium> > keyword > - get all the URL > > However the problem is when i hit the above url in the browser i am able > to get the result for keyword but when i mention this in scraper it shows > me google home page only. > I would really appreciate is someone would help me understand what is > going on? > > Following my code > > from scrapy.spider import BaseSpider > from scrapy.selector import HtmlXPathSelector > from firstScrapper.items import FirstscrapperItem > > class googleSpider(BaseSpider): > name = "googleSpider" > allowed_domains = ["google.co.in"] > start_urls = ["https://www.google.com/?q=selenium/#q=selenium";] > > > def parse(sef, response): > hxs = HtmlXPathSelector(response) > links = hxs.select("//a") > > items = [] > for link in links: > item = FirstscrapperItem() > item["urls"] = link.select("@href").extract() > item["title"] = link.select("text()").extract() > items.append(item) > return items > #print link.select("text()").extract() > # print link.select("@href").extract() > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/groups/opt_out.
