I think you should consider using http://phantomjs.org/ for this, it will
handle a lot of details for you like running javascript on the page that might
include additional content etc that you won't get by just loading it with the
http module. There are a couple of node wrappers, I've used
https://github.com/sgentle/phantomjs-node to write automated testing scripts.
--
Paul Spencer
Sent with Airmail
From: Gaurang shah Gaurang shah
Reply: nodejs@googlegroups.com nodejs@googlegroups.com
Date: December 16, 2013 at 10:51:34 AM
To: nodejs@googlegroups.com nodejs@googlegroups.com
Subject: [nodejs] NodeJS basic Scrappe
Hi guys,
I am trying to create my first web scrapper using NodeJS and Express. What I am
trying to achieve is all the results on the first page of the google search.
However the only problem is, with NodeJs it's not loading page with search
result, however if I visit that page manually it shows me result page.
following is the code, I am not sure what I am doing wrong.
var http = require('https');
var $ = require('cheerio')
var keyword = "selenium";
var options = {
host: 'www.google.co.in',
path: '/?q=selenium/?#q=selenium',
method: 'GET'
};
var request = http.request(options, function (response) {
var body = ''
console.log("guarnag");
response.on("data", function(chunk){
body = body + chunk;
});
response.on("end",function(){
//console.log(body);
var parsedHTML = $.load(body)
parsedHTML('h3.a').map(function(i, link) {
var href = $(link).attr('href')
console.log(href);
})
});
});
request.end();
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to nodejs+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
