I am little new to Java. I am working on web scraping program for school. I
found out before you can web scrape, you have to create a cookie with
userid/password information. I am using HTTPClient library to do that.
Following is my code. I am having problem with httpclient. execute(httpget)
line, it just goes to error.jsp page, without even going to catch section
Please help.
my call: setAuthenticationCookie("","","http://www.msn.com ");
public void setAuthenticationCookie( String sUsername, String sPassword,
String sURL)
{
try{
DefaultHttpClient client = new DefaultHttpClient();
// set per default
//client.getParams().setParameter(
// HttpClientParams.COOKIE_POLICY ,
CookiePolicy.BROWSER_COMPATIBILITY);
HttpGet httpget = new HttpGet(sURL);
HttpResponse response = client.execute(httpget); //PROBLEM HERE
and above too. sounds like client is not getting a valid handler (however
client does not have 0x000 pointer , it does return 213 some number, i guess
that means it points to valid address, coming from C++ background, java
confuses me)
HttpEntity entity = response.getEntity();
if( entity != null){
entity.consumeContent();
}
Cookie[] cookies = client.getState ().getCookies();
HttpPost httppost = new HttpPost(sURL);
NameValuePair[] valuePair = new NameValuePair[] {
new BasicNameValuePair("IDToken1", "username"),
new BasicNameValuePair("IDToken2", "password"),
};
//Only for https
//httppost.setEntity(new UrlEncodedFormEntity(valuePair,
HTTP.UTF_8));
response = client.execute(httppost);
entity = response.getEntity();
if( entity != null){
entity.consumeContent();
}
}
catch (Exception e) {
e.printStackTrace(System.err);
}
}