I think I've understand what the problem is. I've looked closely in chrome-dev-tools to html which is returned by GET request, and it looks like those tables actually are commented in raw html. But then, I suppose, they're uncommented by some JS, depending on display size or user-agent. So, I think I will use some regex magic before feeding request body to Selector to overcome this.
Thank you. On Sunday, September 4, 2016 at 3:28:28 AM UTC+10, Travis Leleu wrote: > > Artem, > > I just pulled up the page source for the site you referenced. It actually > only has 4 <table> elements (of the html style; the other 'tables' on the > page are <div> made to render like a table). > > Your selector is only querying for the 4 actual <table> elements. You > need to identify the selectors for the other data, because it's not in a > table. > > (FWIW, there were 13 <table> elements in the source, but 9 were commented > out.) > > On Fri, Sep 2, 2016 at 8:15 PM, Artem Utin <[email protected] > <javascript:>> wrote: > >> Hello! >> I've tried to parse all tables (there are 6) from page - >> http://www.basketball-reference.com/boxscores/201606190GSW.html . At >> first, I've decided to test selectors using shell. >> So, I've used: >> >> *scrapy shell >> http://www.basketball-reference.com/boxscores/201606190GSW.html >> <http://www.basketball-reference.com/boxscores/201606190GSW.html>* >> *response.selector.css('table')* >> >> >> And got list with only 4 tables, missing first two of them, "Line Score" >> and "Four Factors". I cant select this tables by ids too. >> I've tried this in both py2 and py3 version, also I've imported this page >> as a text and it was not working either. I suspect the issue is in huge >> html comments before this tables. >> >> Can anyone help me with that, please? >> PS. I've wanted to clone repo and test it further, but unable to run tox >> tests with strange error and I've give it up for now. >> >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "scrapy-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/scrapy-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
