Hi, It seems every site has this issue. How did you get around it?
Thanks, On Thursday, October 27, 2016 at 12:35:38 AM UTC-4, ignorant wrote: > > Hi there, > > I am a noob and trying to test this on different product grids. I am not > able to get more than a few (6 to 8) items per page. > > For example, > > import scrapy > > > class NordstromSpider(scrapy.Spider): > name = "nordstrom" > start_urls = [ > ' > http://shop.nordstrom.com/c/womens-dresses-new?origin=leftnav&cm_sp=Top%20Navigation-_-New%20Arrivals > ' > ] > > > def parse(self, response): > for dress in response.css('article.npr-product-module'): > yield { > 'src': dress.css('img.product-photo').xpath('@src'). > extract_first(), > 'url': dress.css('a.product-photo-href').xpath('@href'). > extract_first() > } > > > def noparse(self, response): > page = response.url.split("/")[-2] > filename = 'nordstrom-%s.html' % page > with open(filename, 'wb') as f: > f.write(response.body) > self.log('Saved file %s' % filename) > > > > This gave only 6 items. So I tried another site - > > import scrapy > > > class QuotesSpider(scrapy.Spider): > name = "rtr" > start_urls = [ > 'https://www.renttherunway.com/products/dress' > ] > > > def parse(self, response): > for dress in response.css('div.cycle-image-0'): > yield { > 'image-url': dress.xpath('.//img/@src').extract_first(), > } > > > > This only gave 12 items even though the page has a lot more. > I am guessing that I'm missing a setting somewhere. Any pointers are > appreciated. > > Thanks, > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
