Hello, by default, link extractors in Scrapy do URL normalization or URL canonicalization https://en.wikipedia.org/wiki/URL_normalization
You can tell the link extractor not to do that by passing canonicalize=False >>> lx = scrapy.contrib.linkextractors.LinkExtractor() >>> lx.extract_links(response)[7] Link(url='http://sflib1.sfpl.org:82/search?%2FdBanks+A%2Fdbanks+a%2F1%2C4%2C41%2CE%2F2exact=&1%2C23%2C=&FF=dbanks+american+trust+company', text='Banks American Trust Company', fragment='', nofollow=False) >>> lx = scrapy.contrib.linkextractors.LinkExtractor(canonicalize=False) >>> lx.extract_links(response)[7] Link(url='http://sflib1.sfpl.org:82/search?/dBanks+A/dbanks+a/1%2C4%2C41%2CE/2exact&FF=dbanks+american+trust+company&1%2C23%2C', text='Banks American Trust Company', fragment='', nofollow=False) On Wednesday, November 26, 2014 2:59:00 AM UTC+1, Максимилиан Гомза wrote: > > Have a good day. > > I am new in Scrapy. I am trying to parse photos from links in page > http://sfpl.org/index.php?pg=2000028501 > > XPATH_WAY = '/html/body/div[2]/div[1]/div[3]/div[2]/div[2]/div[2]/ul/li' > XPATH_DETALIED = > ['/html/body/table/tbody/tr[2]/td/table/tbody/tr/td[3]/a[not(@name)]'] > > class SplfDetailsSpider(CrawlSpider): > name = 'splf_details' > rules = [ > Rule(LinkExtractor(allow='search', restrict_xpaths=XPATH_WAY)), > Rule(LinkExtractor(allow='2exact', > restrict_xpaths=XPATH_DETALIED)), > Rule(LinkExtractor(allow='frameset'), callback='detail_parse') > ] > > start_urls = ["http://sfpl.org/index.php?pg=2000028501";] > > Scrapy without problem parse links from the started page, but in next page > when he trying to extract links from XPATH_DETALIED. > There is get a problem. He cuts a part of link, for ecxample: > > From page: > http://sflib1.sfpl.org:82/search/d?SEARCH=Banks+A > > First item in xpath's list: > <a > href="/search?/dBanks+A/dbanks+a/1%2C4%2C41%2CE/frameset&FF=dbanks+american+national+bank&1%2C1%2C">Banks > > American National Bank</a> > > Link Extractor cut a link to: > > http://sflib1.sfpl.org:82//search?/dBanks+A/dbanks+a/1%2C4%2C41%2CE/frameset&FF=dbanks+american+national+bank > > Help please how to resolve this problem. > > > > > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
