Hi Aryé, That's because the website is using Javascript to build most of the page. The elements you want do not appear in the HTML source of the page. You can check by disabling Javascript and reloading the page.
To get the HTML your browser creates with Javascript, you could use a javascript renderer like splash (https://github.com/scrapinghub/splash) or analyse network activity in your browser's Developer Tools to understand where the data in the final page comes from. Hope this helps. /Paul On Tuesday, March 11, 2014 11:04:14 PM UTC+1, Aryé H wrote: > > > > Dear list, > > I am new to scrapy and would greatly appreciate if someone can explain why > the first two selectors in the example below return an empty list ? > I have to go up 2 levels to get a result returned. > > > > > ⟫ scrapy shell http://www.snagfilms.com/films/browse/all/offset/0 > > >>> sel.xpath('//*[@id="rightSectionBrowse"]') > > > > > [] > >>> sel.xpath('/html/body/div[2]/div/div[2]/div[2]/div[2]') > > > > [] > >>> sel.xpath('/html/body/div[2]/div/div[2]') > > > > [<Selector xpath='/html/body/div[2]/div/div[2]' data=u'<div class="module > browse-landing-module'>] > >>> > > > > These xpath expressions that fail to return data are the minimal and > absolute paths to the same HTML element highlighted below in the collapsed > HTML image as copied from firebug. > > > > <https://lh6.googleusercontent.com/-qQKPiFYfuhw/Ux-FobDVZnI/AAAAAAAAIlY/3UVOvHVzsrU/s1600/collapsedHTML.png> > > > Regards, > > Arye. > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
