HttpClient initialization, when/where?
Hi, I will be writing a gateway which will invoke URLs in behalf of several threads and they will return the content. Each thread needs to invoke different URLs with different context (cookies, etc). It looks like right way of doing this is to instantiate the HttpClient once with MultiThreadedHttpConnectionManager and each thread will use httpClient.executeMethod on this HttpClient instance. Or the other alternative is to instantiate one object for each thread, and keep calling executeMethods on them. However, since you can set the state of the HttpClient, I was wondering if I can use the HttpClient for iterative http invocations with different contexts? I mean, is there an in-memory state other than the HttpState that is preserved between these invocations? Would it be enough to create an HttpState and GetMethod for each call, and set it before calling executeMethod? Shortly, what is the optimum mechanism for this functionality? TIA... __ Do you Yahoo!? Yahoo! Movies - Buy advance tickets for 'Shrek 2' http://movies.yahoo.com/showtimes/movie?mid=1808405861 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: HttpClient initialization, when/where?
Hello, you need one HttpState for each thread, since the cookies are stored there. Creating a new state for each request will not work, unless the threads manage the cookies themselves. The HttpClient is associated with a connection pool. If you require independently configured connection pools for each thread, you have to create multiple HttpClient objects. Otherwise, one HttpClient with MultiThreadedHCM is ok. hope that helps, Roland Cabbar Duzayak [EMAIL PROTECTED] 12.05.2004 09:32 Please respond to Commons HttpClient Project To: [EMAIL PROTECTED] cc: Subject:HttpClient initialization, when/where? Hi, I will be writing a gateway which will invoke URLs in behalf of several threads and they will return the content. Each thread needs to invoke different URLs with different context (cookies, etc). It looks like right way of doing this is to instantiate the HttpClient once with MultiThreadedHttpConnectionManager and each thread will use httpClient.executeMethod on this HttpClient instance. Or the other alternative is to instantiate one object for each thread, and keep calling executeMethods on them. However, since you can set the state of the HttpClient, I was wondering if I can use the HttpClient for iterative http invocations with different contexts? I mean, is there an in-memory state other than the HttpState that is preserved between these invocations? Would it be enough to create an HttpState and GetMethod for each call, and set it before calling executeMethod? Shortly, what is the optimum mechanism for this functionality? TIA... __ Do you Yahoo!? Yahoo! Movies - Buy advance tickets for 'Shrek 2' http://movies.yahoo.com/showtimes/movie?mid=1808405861 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: HttpClient initialization, when/where?
Duzayak (I hope I got your name right),
You can create an instance of the HttpState class per thread and pass the respective
HttpState object along with the HTTP method to be executed:
// shared by all the worker threads
HttpConnectionManager connman = new MultiThreadedHttpConnectionManager();
HttpClient agent = new HttpClient(connman);
// one per thread
HttpState httpstate = new HttpState();
// in the worker thread
GetMethod httpget = new GetMethod(http://www.whatever.com;);
try {
agent.executeMethod(null, httpget, httpstate);
// do something useful with the result
} finally {
httpget.releaseConnection();
}
For more details see
http://jakarta.apache.org/commons/httpclient/apidocs/org/apache/commons/httpclient/HttpClient.html#executeMethod(org.apache.commons.httpclient.HostConfiguration,%20org.apache.commons.httpclient.HttpMethod,%20org.apache.commons.httpclient.HttpState)
HTH
Oleg
-Original Message-
From: Cabbar Duzayak [mailto:[EMAIL PROTECTED]
Sent: Wednesday, May 12, 2004 9:33
To: [EMAIL PROTECTED]
Subject: HttpClient initialization, when/where?
Hi,
I will be writing a gateway which will invoke URLs in
behalf of several threads and they will return the
content. Each thread needs to invoke different URLs
with different context (cookies, etc).
It looks like right way of doing this is to
instantiate the HttpClient once with
MultiThreadedHttpConnectionManager and each thread
will use httpClient.executeMethod on this HttpClient
instance. Or the other alternative is to instantiate
one object for each thread, and keep calling
executeMethods on them.
However, since you can set the state of the
HttpClient, I was wondering if I can use the
HttpClient for iterative http invocations with
different contexts? I mean, is there an in-memory
state other than the HttpState that is preserved
between these invocations? Would it be enough to
create an HttpState and GetMethod for each call, and
set it before calling executeMethod?
Shortly, what is the optimum mechanism for this
functionality?
TIA...
__
Do you Yahoo!?
Yahoo! Movies - Buy advance tickets for 'Shrek 2'
http://movies.yahoo.com/showtimes/movie?mid=1808405861
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
***
The information in this email is confidential and may be legally privileged. Access
to this email by anyone other than the intended addressee is unauthorized. If you are
not the intended recipient of this message, any review, disclosure, copying,
distribution, retention, or any action taken or omitted to be taken in reliance on it
is prohibited and may be unlawful. If you are not the intended recipient, please
reply to or forward a copy of this message to the sender and delete the message, any
attachments, and any copies thereof from your system.
***
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
DO NOT REPLY [Bug 28728] - HttpUrl does not accept unescaped passwords
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://issues.apache.org/bugzilla/show_bug.cgi?id=28728. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://issues.apache.org/bugzilla/show_bug.cgi?id=28728 HttpUrl does not accept unescaped passwords [EMAIL PROTECTED] changed: What|Removed |Added Status|RESOLVED|REOPENED Resolution|FIXED | --- Additional Comments From [EMAIL PROTECTED] 2004-05-12 09:04 --- The same applies to HttpsURL. The relevant piece of code is duplicated in these two classes, but it was only fixed in HttpURL. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: HttpClient initialization, when/where?
Can you please elaborate a bit how this connection pool and multithreaded HCM work? Cause, given that httpclient will be executed within my thread, why do we need a multithreaded HCM? Also, connection pool is used for sharing connections between threads? How exactly does it work, what does it improve? Thanks a lot for the information! --- Roland Weber [EMAIL PROTECTED] wrote: Hello, you need one HttpState for each thread, since the cookies are stored there. Creating a new state for each request will not work, unless the threads manage the cookies themselves. The HttpClient is associated with a connection pool. If you require independently configured connection pools for each thread, you have to create multiple HttpClient objects. Otherwise, one HttpClient with MultiThreadedHCM is ok. hope that helps, Roland Cabbar Duzayak [EMAIL PROTECTED] 12.05.2004 09:32 Please respond to Commons HttpClient Project To: [EMAIL PROTECTED] cc: Subject:HttpClient initialization, when/where? Hi, I will be writing a gateway which will invoke URLs in behalf of several threads and they will return the content. Each thread needs to invoke different URLs with different context (cookies, etc). It looks like right way of doing this is to instantiate the HttpClient once with MultiThreadedHttpConnectionManager and each thread will use httpClient.executeMethod on this HttpClient instance. Or the other alternative is to instantiate one object for each thread, and keep calling executeMethods on them. However, since you can set the state of the HttpClient, I was wondering if I can use the HttpClient for iterative http invocations with different contexts? I mean, is there an in-memory state other than the HttpState that is preserved between these invocations? Would it be enough to create an HttpState and GetMethod for each call, and set it before calling executeMethod? Shortly, what is the optimum mechanism for this functionality? TIA... __ Do you Yahoo!? Yahoo! Movies - Buy advance tickets for 'Shrek 2' http://movies.yahoo.com/showtimes/movie?mid=1808405861 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] __ Do you Yahoo!? Yahoo! Movies - Buy advance tickets for 'Shrek 2' http://movies.yahoo.com/showtimes/movie?mid=1808405861 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: HttpClient initialization, when/where?
Hi Duzayak (?), the MultiThreadedHCM is called so because it allows for multiple threads to use the same HttpClient. It has to be used if the application is multithreaded. The connection pool limits the number of simultaneous connections, to a particular host and in general. It keeps resource usage in check. Also, if there are many requests to the same host, open connections may be reused, which avoids the overhead of establishing a new connection for each request. If you want to know how it works, feel free to look at the source code. It's open :-) cheers, Roland Cabbar Duzayak [EMAIL PROTECTED] 12.05.2004 12:26 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc: Subject:Re: HttpClient initialization, when/where? Can you please elaborate a bit how this connection pool and multithreaded HCM work? Cause, given that httpclient will be executed within my thread, why do we need a multithreaded HCM? Also, connection pool is used for sharing connections between threads? How exactly does it work, what does it improve? Thanks a lot for the information!
Cookie Bug?
Hi, I am trying to run the following code for www.google.com: Cookie cookie = initialState.getCookies()[0]; CookieSpec cs = CookiePolicy.getDefaultSpec(); Header h = cs.formatCookieHeader(cookie); Cookie[] cookie2 = cs.parse(cookie.getDomain(), 80, cookie.getPath(), cookie.getSecure(), h); And, it is throwing an exception as: org.apache.commons.httpclient.cookie.MalformedCookieException: Cookie name may not start with $ It looks like h.getValue returns name with $ sign (which was indeed created by the cookiespec), but the same cookiespec can not read this back, because it has a $ sign??? BTW, h.getValue() is: $Version=0; PREF=ID=069e080d47cd4332:TM=1084362072:LM=1084362072:S=0u9G4CTYGPvhxJzn; $Domain=.google.com; $Path=/ Looks like, cookieSpec.parse is expecting the same string without $ signs, in fact, when I remove dollar signs manually, the code works fine. Did I hit the jackpot and found a bug;) Thanks... __ Do you Yahoo!? Yahoo! Movies - Buy advance tickets for 'Shrek 2' http://movies.yahoo.com/showtimes/movie?mid=1808405861 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: HttpClient initialization, when/where?
Well, This information is more than I need;) Appreciate your help. --- Roland Weber [EMAIL PROTECTED] wrote: Hi Duzayak (?), the MultiThreadedHCM is called so because it allows for multiple threads to use the same HttpClient. It has to be used if the application is multithreaded. The connection pool limits the number of simultaneous connections, to a particular host and in general. It keeps resource usage in check. Also, if there are many requests to the same host, open connections may be reused, which avoids the overhead of establishing a new connection for each request. If you want to know how it works, feel free to look at the source code. It's open :-) cheers, Roland Cabbar Duzayak [EMAIL PROTECTED] 12.05.2004 12:26 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc: Subject:Re: HttpClient initialization, when/where? Can you please elaborate a bit how this connection pool and multithreaded HCM work? Cause, given that httpclient will be executed within my thread, why do we need a multithreaded HCM? Also, connection pool is used for sharing connections between threads? How exactly does it work, what does it improve? Thanks a lot for the information! __ Do you Yahoo!? Yahoo! Movies - Buy advance tickets for 'Shrek 2' http://movies.yahoo.com/showtimes/movie?mid=1808405861 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Cookie Bug?
Duzayak, 'Set-Cookie' (request) header is not the same thing as the 'Cookie' (response) header. CookieSpec#formatCookieHeader() method produces a 'Set-Cookie' (request) header, whereas CookieSpec#parse() method is intended to parse 'Cookie' (response) headers I hope this clarifies things a bit Oleg -Original Message- From: Cabbar Duzayak [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 12, 2004 13:47 To: Commons HttpClient Project Subject: Cookie Bug? Hi, I am trying to run the following code for www.google.com: Cookie cookie = initialState.getCookies()[0]; CookieSpec cs = CookiePolicy.getDefaultSpec(); Header h = cs.formatCookieHeader(cookie); Cookie[] cookie2 = cs.parse(cookie.getDomain(), 80, cookie.getPath(), cookie.getSecure(), h); And, it is throwing an exception as: org.apache.commons.httpclient.cookie.MalformedCookieException: Cookie name may not start with $ It looks like h.getValue returns name with $ sign (which was indeed created by the cookiespec), but the same cookiespec can not read this back, because it has a $ sign??? BTW, h.getValue() is: $Version=0; PREF=ID=069e080d47cd4332:TM=1084362072:LM=1084362072:S=0u9G4CTYGPvhxJzn; $Domain=.google.com; $Path=/ Looks like, cookieSpec.parse is expecting the same string without $ signs, in fact, when I remove dollar signs manually, the code works fine. Did I hit the jackpot and found a bug;) Thanks... __ Do you Yahoo!? Yahoo! Movies - Buy advance tickets for 'Shrek 2' http://movies.yahoo.com/showtimes/movie?mid=1808405861 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] *** The information in this email is confidential and may be legally privileged. Access to this email by anyone other than the intended addressee is unauthorized. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system. *** - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
'Socket closed' exception using
Hello, We are trying to use HTTPClient.execute(PostMethod) for sending SOAP requests to Connotate's Web Mining Server (WMS). The requests are sent every 1 minute, and in most cases everything works fine. However, every so often (sometimes way too often!) we are getting the 'Socket closed' exception. The socket timeout was initially set to 10 seconds (1 millis), but even after I set it to 120 seconds (12 millis), we are still seeing it happening quite often. I tried to use both the getResponseBodyAsString() and getResponseBodyAsStream() to retrieve the response, but it does not change anything. The application is running on Windows 2000 (SP4). I tried both the HTTPClient v2.0 (final) and the latest night build (as of yesterday), but the results are the same. We are using HTTP 1.0 (PostMethod.setHttp11(false)), with the application running on the same computer as the WMS. I will appreciate any suggestions about what can be done here. Thank you, Sofya Stack trace: 08:54:37 DEBUG [Thread-19] - enter PostMethod.renerateRequestBody() 08:54:37 DEBUG [Thread-19] - enter EntityEnclosingMethod.renerateRequestBody() 08:54:37 DEBUG [Thread-19] - enter getContentCharSet( Header contentheader ) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parse(String) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parsePair(char[], int, int) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parsePair(char[], int, int) 08:54:37 DEBUG [Thread-19] - Using buffered request body 08:54:37 DEBUG [Thread-19] - enter HttpConnection.getRequestOutputStream() 08:54:37 DEBUG [Thread-19] - Request body sent 08:54:37 DEBUG [Thread-19] - enter HttpConnection.flushRequestOutputStream() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.releaseConnection() 08:54:37 ERROR [Thread-19] - I/O exception executing the get data request 'Execute'. java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.commons.httpclient.HttpConnection$WrappedOutputStream.write(H ttpConnection.java:1368) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:127) at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(Ht tpConnection.java:799) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase .java:2324) at org.apache.commons.httpclient.HttpMethodBase.processRequest(HttpMethodBa se.java:2657) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java :1093) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:6 75) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:5 29) at pbf.travinfo.dcol.CHPDI.AgentLibrary.getSOAPResponse(AgentLibrary.java:6 48) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: 'Socket closed' exception using
How are you setting your headers? That is sometimes the issue on socket closings. At 10:25 AM 5/12/2004, Preygel, Sofya wrote: Hello, We are trying to use HTTPClient.execute(PostMethod) for sending SOAP requests to Connotate's Web Mining Server (WMS). The requests are sent every 1 minute, and in most cases everything works fine. However, every so often (sometimes way too often!) we are getting the 'Socket closed' exception. The socket timeout was initially set to 10 seconds (1 millis), but even after I set it to 120 seconds (12 millis), we are still seeing it happening quite often. I tried to use both the getResponseBodyAsString() and getResponseBodyAsStream() to retrieve the response, but it does not change anything. The application is running on Windows 2000 (SP4). I tried both the HTTPClient v2.0 (final) and the latest night build (as of yesterday), but the results are the same. We are using HTTP 1.0 (PostMethod.setHttp11(false)), with the application running on the same computer as the WMS. I will appreciate any suggestions about what can be done here. Thank you, Sofya Stack trace: 08:54:37 DEBUG [Thread-19] - enter PostMethod.renerateRequestBody() 08:54:37 DEBUG [Thread-19] - enter EntityEnclosingMethod.renerateRequestBody() 08:54:37 DEBUG [Thread-19] - enter getContentCharSet( Header contentheader ) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parse(String) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parsePair(char[], int, int) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parsePair(char[], int, int) 08:54:37 DEBUG [Thread-19] - Using buffered request body 08:54:37 DEBUG [Thread-19] - enter HttpConnection.getRequestOutputStream() 08:54:37 DEBUG [Thread-19] - Request body sent 08:54:37 DEBUG [Thread-19] - enter HttpConnection.flushRequestOutputStream() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.releaseConnection() 08:54:37 ERROR [Thread-19] - I/O exception executing the get data request 'Execute'. java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.commons.httpclient.HttpConnection$WrappedOutputStream.write(H ttpConnection.java:1368) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:127) at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(Ht tpConnection.java:799) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase .java:2324) at org.apache.commons.httpclient.HttpMethodBase.processRequest(HttpMethodBa se.java:2657) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java :1093) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:6 75) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:5 29) at pbf.travinfo.dcol.CHPDI.AgentLibrary.getSOAPResponse(AgentLibrary.java:6 48) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: 'Socket closed' exception using
Here it is: PostMethod post = new PostMethod(m_URL); post.setHttp11(false); post.setRequestHeader(Content-Length, Integer.toString(payload.toString().length())); post.setRequestHeader(Content-type, text/xml;charset=utf-8); post.setRequestHeader(Content-type, text/xml; charset=ISO-8859-1); post.setRequestHeader(SOAPAction, http://tempuri.org/action/Agent.; + action); post.setRequestContentLength(payload.toString().length()); post.setRequestBody(payload.toString()); I tried to also use keep-alive connection option (even though I know it makes little sense for HTTP 1.0), but to no avail. Thanks, Sofya -Original Message- From: Michael McGrady [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 12, 2004 1:29 PM To: Commons HttpClient Project Cc: [EMAIL PROTECTED]; Preygel, Sofya Subject: Re: 'Socket closed' exception using How are you setting your headers? That is sometimes the issue on socket closings. At 10:25 AM 5/12/2004, Preygel, Sofya wrote: Hello, We are trying to use HTTPClient.execute(PostMethod) for sending SOAP requests to Connotate's Web Mining Server (WMS). The requests are sent every 1 minute, and in most cases everything works fine. However, every so often (sometimes way too often!) we are getting the 'Socket closed' exception. The socket timeout was initially set to 10 seconds (1 millis), but even after I set it to 120 seconds (12 millis), we are still seeing it happening quite often. I tried to use both the getResponseBodyAsString() and getResponseBodyAsStream() to retrieve the response, but it does not change anything. The application is running on Windows 2000 (SP4). I tried both the HTTPClient v2.0 (final) and the latest night build (as of yesterday), but the results are the same. We are using HTTP 1.0 (PostMethod.setHttp11(false)), with the application running on the same computer as the WMS. I will appreciate any suggestions about what can be done here. Thank you, Sofya Stack trace: 08:54:37 DEBUG [Thread-19] - enter PostMethod.renerateRequestBody() 08:54:37 DEBUG [Thread-19] - enter EntityEnclosingMethod.renerateRequestBody() 08:54:37 DEBUG [Thread-19] - enter getContentCharSet( Header contentheader ) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parse(String) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parsePair(char[], int, int) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parsePair(char[], int, int) 08:54:37 DEBUG [Thread-19] - Using buffered request body 08:54:37 DEBUG [Thread-19] - enter HttpConnection.getRequestOutputStream() 08:54:37 DEBUG [Thread-19] - Request body sent 08:54:37 DEBUG [Thread-19] - enter HttpConnection.flushRequestOutputStream() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.releaseConnection() 08:54:37 ERROR [Thread-19] - I/O exception executing the get data request 'Execute'. java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.commons.httpclient.HttpConnection$WrappedOutputStream.write(H ttpConnection.java:1368) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:127) at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(Ht
RE: 'Socket closed' exception using
I meant how do you set the response headers? That is probably where the problem is. At 10:32 AM 5/12/2004, Preygel, Sofya wrote: Here it is: PostMethod post = new PostMethod(m_URL); post.setHttp11(false); post.setRequestHeader(Content-Length, Integer.toString(payload.toString().length())); post.setRequestHeader(Content-type, text/xml;charset=utf-8); post.setRequestHeader(Content-type, text/xml; charset=ISO-8859-1); post.setRequestHeader(SOAPAction, http://tempuri.org/action/Agent.; + action); post.setRequestContentLength(payload.toString().length()); post.setRequestBody(payload.toString()); I tried to also use keep-alive connection option (even though I know it makes little sense for HTTP 1.0), but to no avail. Thanks, Sofya -Original Message- From: Michael McGrady [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 12, 2004 1:29 PM To: Commons HttpClient Project Cc: [EMAIL PROTECTED]; Preygel, Sofya Subject: Re: 'Socket closed' exception using How are you setting your headers? That is sometimes the issue on socket closings. At 10:25 AM 5/12/2004, Preygel, Sofya wrote: Hello, We are trying to use HTTPClient.execute(PostMethod) for sending SOAP requests to Connotate's Web Mining Server (WMS). The requests are sent every 1 minute, and in most cases everything works fine. However, every so often (sometimes way too often!) we are getting the 'Socket closed' exception. The socket timeout was initially set to 10 seconds (1 millis), but even after I set it to 120 seconds (12 millis), we are still seeing it happening quite often. I tried to use both the getResponseBodyAsString() and getResponseBodyAsStream() to retrieve the response, but it does not change anything. The application is running on Windows 2000 (SP4). I tried both the HTTPClient v2.0 (final) and the latest night build (as of yesterday), but the results are the same. We are using HTTP 1.0 (PostMethod.setHttp11(false)), with the application running on the same computer as the WMS. I will appreciate any suggestions about what can be done here. Thank you, Sofya Stack trace: 08:54:37 DEBUG [Thread-19] - enter PostMethod.renerateRequestBody() 08:54:37 DEBUG [Thread-19] - enter EntityEnclosingMethod.renerateRequestBody() 08:54:37 DEBUG [Thread-19] - enter getContentCharSet( Header contentheader ) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parse(String) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parsePair(char[], int, int) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parsePair(char[], int, int) 08:54:37 DEBUG [Thread-19] - Using buffered request body 08:54:37 DEBUG [Thread-19] - enter HttpConnection.getRequestOutputStream() 08:54:37 DEBUG [Thread-19] - Request body sent 08:54:37 DEBUG [Thread-19] - enter HttpConnection.flushRequestOutputStream() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.releaseConnection() 08:54:37 ERROR [Thread-19] - I/O exception executing the get data request 'Execute'. java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.commons.httpclient.HttpConnection$WrappedOutputStream.write(H ttpConnection.java:1368) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:127) at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(Ht tpConnection.java:799)
Re: 'Socket closed' exception using
Hi Sofya, There are a couple of possibilities, but it sounds like you are experiencing some connection management issues. If you could, please send some sample code showing how you are using HttpClient, as well as a wire log http://jakarta.apache.org/commons/httpclient/logging.html (just the headers please). Mike Preygel, Sofya wrote: Hello, We are trying to use HTTPClient.execute(PostMethod) for sending SOAP requests to Connotate's Web Mining Server (WMS). The requests are sent every 1 minute, and in most cases everything works fine. However, every so often (sometimes way too often!) we are getting the 'Socket closed' exception. The socket timeout was initially set to 10 seconds (1 millis), but even after I set it to 120 seconds (12 millis), we are still seeing it happening quite often. I tried to use both the getResponseBodyAsString() and getResponseBodyAsStream() to retrieve the response, but it does not change anything. The application is running on Windows 2000 (SP4). I tried both the HTTPClient v2.0 (final) and the latest night build (as of yesterday), but the results are the same. We are using HTTP 1.0 (PostMethod.setHttp11(false)), with the application running on the same computer as the WMS. I will appreciate any suggestions about what can be done here. Thank you, Sofya Stack trace: 08:54:37 DEBUG [Thread-19] - enter PostMethod.renerateRequestBody() 08:54:37 DEBUG [Thread-19] - enter EntityEnclosingMethod.renerateRequestBody() 08:54:37 DEBUG [Thread-19] - enter getContentCharSet( Header contentheader ) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parse(String) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parsePair(char[], int, int) 08:54:37 DEBUG [Thread-19] - enter HeaderElement.parsePair(char[], int, int) 08:54:37 DEBUG [Thread-19] - Using buffered request body 08:54:37 DEBUG [Thread-19] - enter HttpConnection.getRequestOutputStream() 08:54:37 DEBUG [Thread-19] - Request body sent 08:54:37 DEBUG [Thread-19] - enter HttpConnection.flushRequestOutputStream() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.close() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams() 08:54:37 DEBUG [Thread-19] - enter HttpConnection.releaseConnection() 08:54:37 ERROR [Thread-19] - I/O exception executing the get data request 'Execute'. java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.commons.httpclient.HttpConnection$WrappedOutputStream.write(H ttpConnection.java:1368) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:127) at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(Ht tpConnection.java:799) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase .java:2324) at org.apache.commons.httpclient.HttpMethodBase.processRequest(HttpMethodBa se.java:2657) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java :1093) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:6 75) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:5 29) at pbf.travinfo.dcol.CHPDI.AgentLibrary.getSOAPResponse(AgentLibrary.java:6 48) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
@since tags (3.0alpha1 blocker)
We are almost there. Before we can cut the release, though, there's one tedious and laborious task to be taken care of. To help people identify API changes, the new classes and methods must be properly marked with the @since 3.0 tag. We (I in the first place) have been too undisciplined to put @since tags as we hacked away. The pay time is now. I'll deal with the problem in the coming days. A helping hand would be appreciated, though. Oleg - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: 'Socket closed' exception using
Hi Mike,
Here is the code:
PostMethod post = new PostMethod(m_URL);
post.setHttp11(false);
post.setRequestHeader(Content-Length,
Integer.toString(payload.toString().length()));
post.setRequestHeader(Content-type, text/xml;charset=utf-8);
post.setRequestHeader(Content-type, text/xml;
charset=ISO-8859-1);
post.setRequestHeader(SOAPAction,
http://tempuri.org/action/Agent.; + action);
post.setRequestContentLength(payload.toString().length());
post.setRequestBody(payload.toString());
// Log the request (debug mode only)
try
{
m_logger.debug(*** Request: +
post.getRequestBodyAsString());
}
catch (Exception e)
{
}
// Create an instance of HttpClient
HttpClient httpclient = new HttpClient();
// Set the connection and socket timeouts
int requestTimeout = m_socketTimeoutInMillis;
httpclient.setConnectionTimeout(m_connTimeoutInMillis);
httpclient.setTimeout(requestTimeout);
m_logger.info (Setting HTTP timeouts: connection to +
m_connTimeoutInMillis +
millis., socket to + requestTimeout +
millis.);
// Execute the request
int statusCode = -1;
for (int attempt = 1; statusCode == -1 attempt =
m_executeRetries; attempt++)
{
try
{
statusCode = httpclient.executeMethod(post);
}
catch (HttpRecoverableException e)
{
m_logger.debug(Recoverable exception occurred while
executing the +
get data request: + e.getMessage() +
(attempt +
attempt + out of + m_executeRetries +
).);
if
(e.getMessage().startsWith(java.net.SocketTimeoutException: Read timed
out))
{
// Increase the timeout for this request
requestTimeout = requestTimeout * 2;
m_logger.debug(Increasing the request timeout to +
requestTimeout +
millis.);
httpclient.setTimeout(requestTimeout);
}
}
catch (IOException e)
{
m_logger.error(I/O error sending the get data request: +
action + '. , e);
post.releaseConnection();
PBFException ex = new PBFException(I/O error sending the
get data request.);
throw ex;
}
}
// Check if the request succeeded or we just ran out of retry
attempts
if (statusCode == -1)
{
// Request failed. Log the exception, release the connection
and
// throw an exception
m_logger.error(Get data request failed. Unable to recover from
the HTTP +
recoverable exception after +
m_executeRetries + retries.);
post.releaseConnection();
PBFException ex =
new PBFException(Unable to recover from the HTTP
recoverable exception after +
m_executeRetries + retries.);
throw ex;
}
// Request succeeded, get the response )
StringBuffer retval = new StringBuffer();
try
{
InputStream is = post.getResponseBodyAsStream();
if (is != null)
{
ByteArrayOutputStream os = new ByteArrayOutputStream();
byte[] buffer = new byte[8192];
int len;
while ((len = is.read(buffer)) 0)
{
os.write(buffer, 0, len);
}
os.close();
String response = os.toString();
retval = new StringBuffer(response);
// Log the response (debug mode only)
if (m_logger.isDebugEnabled() m_logSOAPCommsFlag)
{
m_logger.debug(*** Response: + decode(response));
}
}
else
{
// Shall see if this ever happens... if yes, will throw an
exception
m_logger.error(No data received (getResponseBodyAsStream()
is null).);
}
}
catch (IOException e)
{
m_logger.error(I/O error receiving the get data response: +
action + '. , e);
PBFException ex = new PBFException(I/O error receiving the get
data response.);
throw ex;
}
finally
{
// Release the current connection to the connection pool
post.releaseConnection();
}
Here are the request headers:
18:04:53 DEBUG [Thread-17] - POST /WS/AgentLibrary.asp
HTTP/1.0[\r][\n]
18:04:53 DEBUG [Thread-17] - Content-Length: 868[\r][\n]
18:04:53 DEBUG [Thread-17] - Content-type: text/xml;
charset=ISO-8859-1[\r][\n]
18:04:53 DEBUG [Thread-17] - SOAPAction:
http://tempuri.org/action/Agent.Execute[\r][\n];
18:04:53 DEBUG [Thread-17] - User-Agent: Jakarta
Commons-HttpClient/2.0final[\r][\n]
18:04:53 DEBUG [Thread-17] - Host: dcgate1[\r][\n]
18:04:53 DEBUG [Thread-17] - [\r][\n]
And this are the
RE: 'Socket closed' exception using
Thank you for your response, Oleg.
1. The application development started when the server application supported only
HTTP1.0.
2. The application is multi-threaded, but all HTTP requests are issued from a single
thread. It is not the main thread, though, which I hope makes no difference as this is
where all the HTTPClient objects are created.
3. I have contacted Connotate tech support and hope they will be able to answer some
of the guestions about what is happening on on the server side.
What I cannot understand, though, is this fragment of the log file:
08:54:37 DEBUG [Thread-19] - Request body sent
08:54:37 DEBUG [Thread-19] - enter .flushRequestOutputStream()
08:54:37 DEBUG [Thread-19] - enter HttpConnection.close()
08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams()
08:54:37 DEBUG [Thread-19] - enter HttpConnection.close()
08:54:37 DEBUG [Thread-19] - enter HttpConnection.closeSockedAndStreams()
08:54:37 DEBUG [Thread-19] - enter HttpConnection.releaseConnection()
Does not this prove that the socket is closed from the application side, i.e from
inside the HTTPClient? Or it is possible for the HttpConnection.close() to be called
when a closed socket is detected (to clean up the resources, etc.)?
Thank you!
Sofya
-Original Message-
From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED]
Sent: Wed 5/12/2004 6:41 PM
To: Commons HttpClient Project
Cc:
Subject: RE: 'Socket closed' exception using
Sofya,
(1) What's the reason for using HTTP/1.0 if the target server is clearly
capable of doing HTTP/1.1?
18:04:53 DEBUG [Thread-17] - POST /WS/AgentLibrary.asp
HTTP/1.0[\r][\n]
(2) Are you executing HTTP requests using multiple threads by any
chance?
(3) 'Socket closed' IO exception is usually caused when the target
server drops connection on the unsuspecting HttpClient while it is still
in the process of sending request or reading response. The problem you
have been experiencing in fact _may_ well be on the server side. If you
have access to the target server you may want to examine the server
logs.
In your case the connection to the server gets dropped as soon as
HttpClient is done sending the request body (to be more precise while
flushing the output buffer). That makes me believe that the server is
the most likely culprit, not HttpClient
Oleg
On Thu, 2004-05-13 at 00:09, Preygel, Sofya wrote:
Hi Mike,
Here is the code:
PostMethod post = new PostMethod(m_URL);
post.setHttp11(false);
post.setRequestHeader(Content-Length,
Integer.toString(payload.toString().length()));
post.setRequestHeader(Content-type, text/xml;charset=utf-8);
post.setRequestHeader(Content-type, text/xml;
charset=ISO-8859-1);
post.setRequestHeader(SOAPAction,
http://tempuri.org/action/Agent.; + action);
post.setRequestContentLength(payload.toString().length());
post.setRequestBody(payload.toString());
// Log the request (debug mode only)
try
{
m_logger.debug(*** Request: +
post.getRequestBodyAsString());
}
catch (Exception e)
{
}
// Create an instance of HttpClient
HttpClient httpclient = new HttpClient();
// Set the connection and socket timeouts
int requestTimeout = m_socketTimeoutInMillis;
httpclient.setConnectionTimeout(m_connTimeoutInMillis);
httpclient.setTimeout(requestTimeout);
m_logger.info (Setting HTTP timeouts: connection to +
m_connTimeoutInMillis +
millis., socket to + requestTimeout +
millis.);
// Execute the request
int statusCode = -1;
for (int attempt = 1; statusCode == -1 attempt =
m_executeRetries; attempt++)
{
try
{
statusCode = httpclient.executeMethod(post);
}
catch (HttpRecoverableException e)
{
m_logger.debug(Recoverable exception occurred while
executing the +
get data request: + e.getMessage() +
(attempt +
attempt + out of + m_executeRetries +
).);
if
Re: 'Socket closed' exception using
Hi Sofya, I agree with Oleg, it seems that the server is closing the connection in the middle of a request. Does not this prove that the socket is closed from the application side, i.e from inside the HTTPClient? Or it is possible for the HttpConnection.close() to be called when a closed socket is detected (to clean up the resources, etc.)? Yes, HttpClient closes the connection when it encounters the Exception while writing the request. see http://jakarta.apache.org/commons/httpclient/xref/org/apache/commons/ httpclient/HttpMethodBase.html#2657 Mike - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: @since tags (3.0alpha1 blocker)
Docs updated. Mike On May 12, 2004, at 5:53 PM, Oleg Kalnichevski wrote: We are almost there. Before we can cut the release, though, there's one tedious and laborious task to be taken care of. To help people identify API changes, the new classes and methods must be properly marked with the @since 3.0 tag. We (I in the first place) have been too undisciplined to put @since tags as we hacked away. The pay time is now. I'll deal with the problem in the coming days. A helping hand would be appreciated, though. Oleg - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Persistent HTTPS connections
Using HttpClient 2.0
JDK 1.4.2_04 on Fedora
Is there anything special that I have to do to make use of persistent
HTTP(S) connections with HttpClient other than using
MultiThreadedHttpConnectionManager ?
Basically, what I am doing is the following ( more explanation after the
snippet of the source code ):
MultiThreadedHttpConnectionManager connectionManager =
new MultiThreadedHttpConnectionManager();
connectionManager.setConnectionStaleCheckingEnabled( true );
connectionManager.setMaxConnectionsPerHost( 10 );
connectionManager.setMaxTotalConnections( 100 );
this.httpClient = new HttpClient( connectionManager );
this.httpClient.setConnectionTimeout( 3 );
this.httpClient.setTimeout( 3 );
and then within a thread, the thread does this:
String content;
HttpMethod method;
try {
method = new PostMethod( connectionURL );
method.setDoAuthentication( true );
method.setRequestHeader( Content-Type, contentType + ;
charset= + this.outboundEncoding);
if( method instanceof PostMethod ) {
PostMethod postMethod = (PostMethod) method;
postMethod.setRequestBody( content );
}
int responseCode = this.httpClient.executeMethod( method );
String response = method.getResponseBodyAsString();
.
}
catch( Exception ex ) {}
finally {
if( method != null ) method.releaseConnection();
}
What I am doing, then, from a JUnit test class, is:
1) Send one HTTP POST to a URL, which works and I get the response.
2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30
seconds )
3) Send another HTTP POST to the same URL.
What I am seeing with ethereal is that, after 30 seconds of no activity
( no TCP ACKs whatever on the socket ), the web server sends a a TLS alert.
So what actually happens is this:
1) Send one HTTP POST to a URL, which works and I get the response.
2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30
seconds )
3) Web server sends a TLS alert after 30 seconds of inactivity. Web
server sends a TCP FIN, Java sends a TCP ACK _only_ .
4) Wake up from sleep at the 40 second mark, send another HTTP POST to
the same URL.
5) Receive a TLS alert instead of a HTTP response. The TLS alert
according to JSSE's debug mode is:
main, RECV TLSv1 ALERT: warning, close_notify
main, called closeInternal(false)
main, SEND TLSv1 ALERT: warning, description = close_notify
6) HttpClient throws an HttpRecoverableException, shown below:
org.apache.commons.httpclient.HttpRecoverableException:
org.apache.commons.httpclient.HttpRecoverableException: Error in parsing
the status line from the response: unable to find line starting with HTTP
Question is, was I correct in initially assuming that
MultiThreadedHttpConnectionManager should have handled this case ?
e.g... .detected that the exception, and retried the HTTP POST by
creating a new HTTPS socket ?
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
