Short answer: The shipping time is generated through javascript. Long answer: Did you use the xpath straight from firebug/chrome devtools? Because those tools show you the xpath of the processed html including any dom modification made by javascript. That will not work in scrapy as it doesn't render javascript by itself.
The hard way to extract that value is to find the javascript code that modifies that element, which is around line 782, a XHR request to a QueryStockStatus api. So you should replicate that request to get the stock status and shipping date. The other alternative is to render the javascript through splash or something similar. Regards, Rolando On Thu, Dec 19, 2013 at 10:17 AM, Paul <[email protected]> wrote: > > hi, this is an example link that I have run in the scrapy shell > > http://www.focalprice.com/HL1009X/Sports_Polarized_Sunglasses_Black_Orange.html > > I have been trying to scrape the stock status text i.e. 'in Stock - > Shipping within 24 hours' however Scrapy does not return anything? > > I have tried > sel.xpath('//*[@class="cgreen"]/text()').extract() > sel.xpath('//*[@id="shippingtime"]').extract() > > even when i use the full path: > sel.xpath('//body/div[5]/div[2]/div[2]/div[3]/p').extract() > for some reason it still skips the 'in Stock - Shipping within 24 hours' > part > > this is odd because csgreen class is not even shown anywhere when > executing the full path however mr10 and cblue do exist! > i.e. if you do: sel.xpath('//*[@class="mr10"]/text()').extract() you get > the worldwide free shipping text > How is it that csgreen doesn't exist? > > is it common for Scrapy to miss things or am I doing something wrong??? > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/groups/opt_out.
