Re: Why is POST (HttpClient) is acting like this?
On Tue, 2007-11-20 at 23:15 -0800, terry_513 wrote: Hello, In my desktop application, using HttpClient 3.1, I am navigating a web site. I am trying to retreive the site https://edit.yahoo.com/registration?.intl=usnew=1.done=http, I pass the respective parameters and perform a POST. The post requires redirect. So, I get the Location from the header which points to http://yahoo.com; and get the new location url via GET method. My problem is: the submission if form should be going somehwere else, but it is going to the home page. Why so? Where I may be going wrong? The code doesn't throw any exception or error. But the page that I receive is not expected. Any idea, guidance will be of great help. Thanks This has nothing to do with HttpClient. Yahoo and many other high profile sites are known to employ various techniques to detect and prevent automated screen-scraping. Oleg - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Why is POST (HttpClient) is acting like this?
Thanks Oleg. After some Rnd, now the same page is coming on post response. Their are programs that work through such sites via desktop application. What am I lacking, that I get wrong results? Their must be something to work with these sites also. Any idea or suggestion would be of great help. If you want, I can pass you the code. Please help me. I see in this site, you and other experts have helped so many people. Please. Thanks olegk wrote: On Tue, 2007-11-20 at 23:15 -0800, terry_513 wrote: Hello, In my desktop application, using HttpClient 3.1, I am navigating a web site. I am trying to retreive the site https://edit.yahoo.com/registration?.intl=usnew=1.done=http, I pass the respective parameters and perform a POST. The post requires redirect. So, I get the Location from the header which points to http://yahoo.com; and get the new location url via GET method. My problem is: the submission if form should be going somehwere else, but it is going to the home page. Why so? Where I may be going wrong? The code doesn't throw any exception or error. But the page that I receive is not expected. Any idea, guidance will be of great help. Thanks This has nothing to do with HttpClient. Yahoo and many other high profile sites are known to employ various techniques to detect and prevent automated screen-scraping. Oleg - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Why-is-POST-%28HttpClient%29-is-acting-like-this--tf4848502.html#a13875050 Sent from the HttpClient-User mailing list archive at Nabble.com. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Why is POST (HttpClient) is acting like this?
Its a desktop application. I have set Full wire + context Log. At present,
the post response comes on the same page.
[code]
client = new HttpClient();
//client.getParams().setParameter(http.protocol.version,
HttpVersion.HTTP_1_1);
client.getHostConfiguration().setHost(ymailSinupUrl, 443, https);
connectMgr = client.getHttpConnectionManager();
hc = client.getHostConfiguration();
hc.getParams().setParameter(http.protocol.version,
HttpVersion.HTTP_1_0);
client.getParams().setCookiePolicy(null);
..
NameValuePair data[] = {
new NameValuePair(u, _dsh), new NameValuePair(t, _t1),
new NameValuePair(preferredcontent, this.preferLang), new
NameValuePair(firstname, FirstName), new NameValuePair(secondname,
LastName), new NameValuePair(gender, this.gender), new
NameValuePair(mm, this.bMm), new NameValuePair(dd, bDate), new
NameValuePair(, bYyyy), new NameValuePair(country, loc), new
NameValuePair(postalcode, postalCode),
new NameValuePair(yahooid, Email), new
NameValuePair(password, Passwrd), new NameValuePair(passwordconfirm,
passwrdAgain),
new NameValuePair(altemail, this.alterEmail), new
NameValuePair(secquestion, selection), new
NameValuePair(secquestionanswer, this.IdentifyAnswer), new
NameValuePair(cword, newaccountcaptcha), new NameValuePair(cdata,
this._continue), new NameValuePair(tos_agreed, this.doAgree), new
NameValuePair(IAgreeBtn, submitbutton), new NameValuePair(action,
formAction)
};
...
public String POST(String url, NameValuePair[] data) {
String res = ;
String URL = ymailSinupUrl + _ylt=A9FJpMBjCkRHERcBDgCZ2PAI;
PostMethod method = new PostMethod(URL);
// SET PROPERTIES
method.getParams().setParameter(http.method.retry-handler, new
DefaultHttpMethodRetryHandler(3, false));
method.setRequestHeader(User-agent, USER_AGENT);
method.setRequestHeader(Accept, ACCEPT);
method.setRequestHeader(Accept-Language, ACCEPT_LANG);
method.setRequestHeader(Accept-Charset, ACCEPT_CHAR);
method.setRequestHeader(Keep-Alive, KEEP_ALIVE);
method.setRequestHeader(Connection, CONNECTION);
method.setRequestHeader(Content-Type, text/html);
if (! .equals(lastUrl))
method.setRequestHeader(Referer, lastUrl);
if (connectMgr.getConnection(hc).isOpen() == true)
System.out.println(Connection is OPEN);
else {
System.out.println(Connection is *** NOT OPEN
*** );
}
method.setRequestBody(data);
int statusCode = 0;
try {
statusCode = client.executeMethod(hc, method);
System.out.println(Register Send: +
method.getStatusLine().toString());
}catch (HttpException e) {
method.releaseConnection();
System.out.println(HTTP EXception : + e.getMessage());
}catch (IOException ie) {
method.releaseConnection();
System.out.println(Error Exe Method - Post. Status Code = +
statusCode);
ie.printStackTrace();
}
if (statusCode != 200) {
System.err.println((new StringBuilder()).append(POST Method
failed: ).append(method.getStatusLine()).toString());
} // else {
InputStream inputStream = null;
BufferedReader input = null;
try {
inputStream = method.getResponseBodyAsStream();
input = new BufferedReader(new
InputStreamReader(inputStream));
String str;
while((str = input.readLine()) != null) {
res = (new
StringBuilder()).append(res).append(str).toString();
}
input.close();
} catch (IOException ie) {
method.releaseConnection();
ie.printStackTrace();
}
//}
try {
lastUrl = method.getURI().toString();
} catch (HttpException he) {
he.printStackTrace();
}finally {
method.releaseConnection();
}
System.out.println((new StringBuilder()).append(Status Code =
).append(statusCode).toString());
Header h;
if (statusCode == 302) {
h = method.getResponseHeader(Location);
System.out.println(Header = + h.getValue());
// REQUIRES RE-DIRECT
System.out.println(RE DIRECTING TARGET .);
res = GET(h.getValue());
}
return res;
}
[/code]
in NameValuePairs,
tos_agreed is a checkbox I have assigned it 1. IAgreeBtn, has the same
value as the html source has - Create My Account, action has the
formAction url. Other all parameters r proper, captcha is also entered
correctly. u, t r parsed from html source taken
Re: Why is POST (HttpClient) is acting like this?
So what does a comparison of the WireShark logs show you?
On 21/11/2007, terry_513 [EMAIL PROTECTED] wrote:
Its a desktop application. I have set Full wire + context Log. At present,
the post response comes on the same page.
[code]
client = new HttpClient();
//client.getParams().setParameter(http.protocol.version,
HttpVersion.HTTP_1_1);
client.getHostConfiguration().setHost(ymailSinupUrl, 443, https);
connectMgr = client.getHttpConnectionManager();
hc = client.getHostConfiguration();
hc.getParams().setParameter(http.protocol.version,
HttpVersion.HTTP_1_0);
client.getParams().setCookiePolicy(null);
..
NameValuePair data[] = {
new NameValuePair(u, _dsh), new NameValuePair(t, _t1),
new NameValuePair(preferredcontent, this.preferLang), new
NameValuePair(firstname, FirstName), new NameValuePair(secondname,
LastName), new NameValuePair(gender, this.gender), new
NameValuePair(mm, this.bMm), new NameValuePair(dd, bDate), new
NameValuePair(, bYyyy), new NameValuePair(country, loc), new
NameValuePair(postalcode, postalCode),
new NameValuePair(yahooid, Email), new
NameValuePair(password, Passwrd), new NameValuePair(passwordconfirm,
passwrdAgain),
new NameValuePair(altemail, this.alterEmail), new
NameValuePair(secquestion, selection), new
NameValuePair(secquestionanswer, this.IdentifyAnswer), new
NameValuePair(cword, newaccountcaptcha), new NameValuePair(cdata,
this._continue), new NameValuePair(tos_agreed, this.doAgree), new
NameValuePair(IAgreeBtn, submitbutton), new NameValuePair(action,
formAction)
};
...
public String POST(String url, NameValuePair[] data) {
String res = ;
String URL = ymailSinupUrl + _ylt=A9FJpMBjCkRHERcBDgCZ2PAI;
PostMethod method = new PostMethod(URL);
// SET PROPERTIES
method.getParams().setParameter(http.method.retry-handler, new
DefaultHttpMethodRetryHandler(3, false));
method.setRequestHeader(User-agent, USER_AGENT);
method.setRequestHeader(Accept, ACCEPT);
method.setRequestHeader(Accept-Language, ACCEPT_LANG);
method.setRequestHeader(Accept-Charset, ACCEPT_CHAR);
method.setRequestHeader(Keep-Alive, KEEP_ALIVE);
method.setRequestHeader(Connection, CONNECTION);
method.setRequestHeader(Content-Type, text/html);
if (! .equals(lastUrl))
method.setRequestHeader(Referer, lastUrl);
if (connectMgr.getConnection(hc).isOpen() == true)
System.out.println(Connection is OPEN);
else {
System.out.println(Connection is *** NOT OPEN
*** );
}
method.setRequestBody(data);
int statusCode = 0;
try {
statusCode = client.executeMethod(hc, method);
System.out.println(Register Send: +
method.getStatusLine().toString());
}catch (HttpException e) {
method.releaseConnection();
System.out.println(HTTP EXception : + e.getMessage());
}catch (IOException ie) {
method.releaseConnection();
System.out.println(Error Exe Method - Post. Status Code = +
statusCode);
ie.printStackTrace();
}
if (statusCode != 200) {
System.err.println((new StringBuilder()).append(POST Method
failed: ).append(method.getStatusLine()).toString());
} // else {
InputStream inputStream = null;
BufferedReader input = null;
try {
inputStream = method.getResponseBodyAsStream();
input = new BufferedReader(new
InputStreamReader(inputStream));
String str;
while((str = input.readLine()) != null) {
res = (new
StringBuilder()).append(res).append(str).toString();
}
input.close();
} catch (IOException ie) {
method.releaseConnection();
ie.printStackTrace();
}
//}
try {
lastUrl = method.getURI().toString();
} catch (HttpException he) {
he.printStackTrace();
}finally {
method.releaseConnection();
}
System.out.println((new StringBuilder()).append(Status Code =
).append(statusCode).toString());
Header h;
if (statusCode == 302) {
h = method.getResponseHeader(Location);
System.out.println(Header = + h.getValue());
// REQUIRES RE-DIRECT
System.out.println(RE DIRECTING TARGET .);
res = GET(h.getValue());
}
return res;
}
[/code]
in NameValuePairs,
tos_agreed is a checkbox I have assigned it 1. IAgreeBtn, has the same
value as the html source has - Create My Account, action has the
formAction url. Other all parameters r proper,
Re: Why is POST (HttpClient) is acting like this?
Sebb, How do I use WireShark to get the info? I have never used before. Can you guide me, please. Thanks -- View this message in context: http://www.nabble.com/Why-is-POST-%28HttpClient%29-is-acting-like-this--tf4848502.html#a13881292 Sent from the HttpClient-User mailing list archive at Nabble.com. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Why is POST (HttpClient) is acting like this?
The web site, i am working with is https://edit.yahoo.com/registration?.intl=usnew=1.done=http -- View this message in context: http://www.nabble.com/Why-is-POST-%28HttpClient%29-is-acting-like-this--tf4848502.html#a13881716 Sent from the HttpClient-User mailing list archive at Nabble.com. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
