Hi Guys, I have just started with the scrapy and facing a problem. Let me first tell you what i am trying to develope.
- search for the keyword on the google. - https://www.google.com/?q=keyword/#q=<https://www.google.com/?q=selenium/#q=selenium> keyword - get all the URL However the problem is when i hit the above url in the browser i am able to get the result for keyword but when i mention this in scraper it shows me google home page only. I would really appreciate is someone would help me understand what is going on? Following my code from scrapy.spider import BaseSpider from scrapy.selector import HtmlXPathSelector from firstScrapper.items import FirstscrapperItem class googleSpider(BaseSpider): name = "googleSpider" allowed_domains = ["google.co.in"] start_urls = ["https://www.google.com/?q=selenium/#q=selenium";] def parse(sef, response): hxs = HtmlXPathSelector(response) links = hxs.select("//a") items = [] for link in links: item = FirstscrapperItem() item["urls"] = link.select("@href").extract() item["title"] = link.select("text()").extract() items.append(item) return items #print link.select("text()").extract() # print link.select("@href").extract() -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/groups/opt_out.
