I usually look at the ajax calls made from the page to see if I can't just mock that request and get the data back right from the server. (Note it's possible the site is sending the product name via the original html page, and just uses js to swap it out -- in that case, I would recommend selenium w/ phantomjs [only because it was easy for me to setup, and just worked for me]).
That'll take up to 10x the resources of just a basic http request, but it's still generally minimal unless you're scraping millions of pages. On Sat, Nov 22, 2014 at 8:39 AM, JEBI93 <[email protected]> wrote: > Yep thats what i got to, whats easiest way to scrape it, Selenium, > Phantom.js? I don't have expirience with scraping JavaScript. > > субота, 22. новембар 2014. 17.35.17 UTC+1, Travis Leleu је написао/ла: >> >> Yes, it's a piece of javascript loading the h1 value. When I load the >> page w/o js enabled, it just reads " >> Proizvod >> " >> >> >> >> On Sat, Nov 22, 2014 at 8:26 AM, JEBI93 <[email protected]> wrote: >> >>> http://emmi.rs/procesori-%28cpu%29-cpu-fm2-amd-athlon% >>> E2%84%A2-x4-quad-core-750k.11.html?productId=29644 is link. >>> Here is screenshot of what i would like to scrape(content is same as you >>> can see):http://i.imgur.com/QYW5Cgp.png. >>> The lower one(orange h1) just prints Product, maybe there's some >>> javascript code running in background or something I can't figure it out. >>> Price, description and everything else works just fine. >>> >>> субота, 22. новембар 2014. 17.05.50 UTC+1, Travis Leleu је написао/ла: >>>> >>>> You didn't provide a link to the site in question, but how about >>>> trying something like this? (I often mix CSS and XPath for convenience.) >>>> >>>> sel.css( 'div#blinks' )[0].xpath( './/text()' ) >>>> >>>> (I think the circled text is in the div's text node -- if it's in the >>>> span, just make the xpath selector start with the span. >>>> >>>> >>>> On Sat, Nov 22, 2014 at 6:35 AM, JEBI93 <[email protected]> >>>> wrote: >>>> >>>>> So i was scraping ecommerce site and i can't figure which Xpath to use >>>>> in this div tag:http://i.imgur.com/ltuVfTQ.png >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "scrapy-users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/scrapy-users. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "scrapy-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/scrapy-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
