Re: Persistent HTTPS connections
John No reason other than HttpMethod & HttpMethodBase being horrible monstrosities badly in need of a compete redesign Oleg On Fri, 2004-05-14 at 09:02, Jesus M. Salvo Jr. wrote: > One small thing though. Why is setMethodRetryHandler() defined in > HttpMethodBase class instead of the HttpMethod interface ? > > > Jesus M. Salvo Jr. wrote: > > > > > Hi Oleg, > > > > Thanks for that. I have now implemented my own MethodRetryHandler and > > got rid of my custom loop / retry. > > Cleaner log files for my application as well. > > > > > > John > > > > Kalnichevski, Oleg wrote: > > > >> John, > >> > >> Please correct me if I am wrong (which may well be the case) > >> SO_TIMEOUT only affects socket read operations. I thought it had > >> nothing to do with SSL inactivity timeout. But it looks like it might. > >> > >> There's another way to deal with recoverable exceptions. You can > >> provide a custom implementation of MethodRetryHandler > >> > >> http://jakarta.apache.org/commons/httpclient/apidocs/org/apache/commons/httpclient/MethodRetryHandler.html > >> > >> > >> The default implementation of the MethodRetryHandler is quite > >> conservative. It does not allow the method to be retried if the > >> request has been transmitted in its entirety. > >> http://jakarta.apache.org/commons/httpclient/xref/org/apache/commons/httpclient/DefaultMethodRetryHandler.html#62 > >> > >> > >> Oleg > >> > >> > >> -Original Message- > >> From: Jesus M. Salvo Jr. [mailto:[EMAIL PROTECTED] > >> Sent: Thursday, May 13, 2004 7:40 > >> To: [EMAIL PROTECTED] > >> Subject: Persistent HTTPS connections > >> > >> > >> > >> Using HttpClient 2.0 > >> JDK 1.4.2_04 on Fedora > >> > >> Is there anything special that I have to do to make use of persistent > >> HTTP(S) connections with HttpClient other than using > >> MultiThreadedHttpConnectionManager ? > >> > >> Basically, what I am doing is the following ( more explanation after the > >> snippet of the source code ): > >> > >>MultiThreadedHttpConnectionManager connectionManager = > >> new MultiThreadedHttpConnectionManager(); > >>connectionManager.setConnectionStaleCheckingEnabled( true ); > >>connectionManager.setMaxConnectionsPerHost( 10 ); > >>connectionManager.setMaxTotalConnections( 100 ); > >> > >>this.httpClient = new HttpClient( connectionManager ); > >>this.httpClient.setConnectionTimeout( 3 ); > >>this.httpClient.setTimeout( 3 ); > >> > >> and then within a thread, the thread does this: > >> > >> String content; > >> HttpMethod method; > >> > >>try { > >> method = new PostMethod( connectionURL ); > >>method.setDoAuthentication( true ); > >> method.setRequestHeader( "Content-Type", contentType + "; > >> charset=" + this.outboundEncoding); > >> if( method instanceof PostMethod ) { > >>PostMethod postMethod = (PostMethod) method; > >>postMethod.setRequestBody( content ); > >> } > >> int responseCode = this.httpClient.executeMethod( method ); > >> String response = method.getResponseBodyAsString(); > >> . > >>} > >>catch( Exception ex ) {} > >>finally { > >> if( method != null ) method.releaseConnection(); > >>} > >> > >> > >> What I am doing, then, from a JUnit test class, is: > >> > >> 1) Send one HTTP POST to a URL, which works and I get the response. > >> 2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30 > >> seconds ) > >> 3) Send another HTTP POST to the same URL. > >> > >> What I am seeing with ethereal is that, after 30 seconds of no activity > >> ( no TCP ACKs whatever on the socket ), the web server sends a a TLS > >> alert. > >> So what actually happens is this: > >> > >> > >> 1) Send one HTTP POST to a URL, which works and I get the response. > >> 2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30 > >> seconds ) > >> 3) Web server sends a TLS alert after 30 seconds of inactivity. Web > >> server sends a TCP FIN, Java se
Re: Persistent HTTPS connections
One small thing though. Why is setMethodRetryHandler() defined in HttpMethodBase class instead of the HttpMethod interface ? Jesus M. Salvo Jr. wrote: Hi Oleg, Thanks for that. I have now implemented my own MethodRetryHandler and got rid of my custom loop / retry. Cleaner log files for my application as well. John Kalnichevski, Oleg wrote: John, Please correct me if I am wrong (which may well be the case) SO_TIMEOUT only affects socket read operations. I thought it had nothing to do with SSL inactivity timeout. But it looks like it might. There's another way to deal with recoverable exceptions. You can provide a custom implementation of MethodRetryHandler http://jakarta.apache.org/commons/httpclient/apidocs/org/apache/commons/httpclient/MethodRetryHandler.html The default implementation of the MethodRetryHandler is quite conservative. It does not allow the method to be retried if the request has been transmitted in its entirety. http://jakarta.apache.org/commons/httpclient/xref/org/apache/commons/httpclient/DefaultMethodRetryHandler.html#62 Oleg -Original Message- From: Jesus M. Salvo Jr. [mailto:[EMAIL PROTECTED] Sent: Thursday, May 13, 2004 7:40 To: [EMAIL PROTECTED] Subject: Persistent HTTPS connections Using HttpClient 2.0 JDK 1.4.2_04 on Fedora Is there anything special that I have to do to make use of persistent HTTP(S) connections with HttpClient other than using MultiThreadedHttpConnectionManager ? Basically, what I am doing is the following ( more explanation after the snippet of the source code ): MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager(); connectionManager.setConnectionStaleCheckingEnabled( true ); connectionManager.setMaxConnectionsPerHost( 10 ); connectionManager.setMaxTotalConnections( 100 ); this.httpClient = new HttpClient( connectionManager ); this.httpClient.setConnectionTimeout( 3 ); this.httpClient.setTimeout( 3 ); and then within a thread, the thread does this: String content; HttpMethod method; try { method = new PostMethod( connectionURL ); method.setDoAuthentication( true ); method.setRequestHeader( "Content-Type", contentType + "; charset=" + this.outboundEncoding); if( method instanceof PostMethod ) { PostMethod postMethod = (PostMethod) method; postMethod.setRequestBody( content ); } int responseCode = this.httpClient.executeMethod( method ); String response = method.getResponseBodyAsString(); . } catch( Exception ex ) {} finally { if( method != null ) method.releaseConnection(); } What I am doing, then, from a JUnit test class, is: 1) Send one HTTP POST to a URL, which works and I get the response. 2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30 seconds ) 3) Send another HTTP POST to the same URL. What I am seeing with ethereal is that, after 30 seconds of no activity ( no TCP ACKs whatever on the socket ), the web server sends a a TLS alert. So what actually happens is this: 1) Send one HTTP POST to a URL, which works and I get the response. 2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30 seconds ) 3) Web server sends a TLS alert after 30 seconds of inactivity. Web server sends a TCP FIN, Java sends a TCP ACK _only_ . 4) Wake up from sleep at the 40 second mark, send another HTTP POST to the same URL. 5) Receive a TLS alert instead of a HTTP response. The TLS alert according to JSSE's debug mode is: main, RECV TLSv1 ALERT: warning, close_notify main, called closeInternal(false) main, SEND TLSv1 ALERT: warning, description = close_notify 6) HttpClient throws an HttpRecoverableException, shown below: org.apache.commons.httpclient.HttpRecoverableException: org.apache.commons.httpclient.HttpRecoverableException: Error in parsing the status line from the response: unable to find line starting with "HTTP" Question is, was I correct in initially assuming that MultiThreadedHttpConnectionManager should have "handled" this case ? e.g... .detected that the exception, and retried the HTTP POST by creating a new HTTPS socket ? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] *** The information in this email is confidential and may be legally privileged. Access to this email by anyone other than the intended addressee is unauthorized. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of
Re: Persistent HTTPS connections
Hi Oleg, Thanks for that. I have now implemented my own MethodRetryHandler and got rid of my custom loop / retry. Cleaner log files for my application as well. John Kalnichevski, Oleg wrote: John, Please correct me if I am wrong (which may well be the case) SO_TIMEOUT only affects socket read operations. I thought it had nothing to do with SSL inactivity timeout. But it looks like it might. There's another way to deal with recoverable exceptions. You can provide a custom implementation of MethodRetryHandler http://jakarta.apache.org/commons/httpclient/apidocs/org/apache/commons/httpclient/MethodRetryHandler.html The default implementation of the MethodRetryHandler is quite conservative. It does not allow the method to be retried if the request has been transmitted in its entirety. http://jakarta.apache.org/commons/httpclient/xref/org/apache/commons/httpclient/DefaultMethodRetryHandler.html#62 Oleg -Original Message- From: Jesus M. Salvo Jr. [mailto:[EMAIL PROTECTED] Sent: Thursday, May 13, 2004 7:40 To: [EMAIL PROTECTED] Subject: Persistent HTTPS connections Using HttpClient 2.0 JDK 1.4.2_04 on Fedora Is there anything special that I have to do to make use of persistent HTTP(S) connections with HttpClient other than using MultiThreadedHttpConnectionManager ? Basically, what I am doing is the following ( more explanation after the snippet of the source code ): MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager(); connectionManager.setConnectionStaleCheckingEnabled( true ); connectionManager.setMaxConnectionsPerHost( 10 ); connectionManager.setMaxTotalConnections( 100 ); this.httpClient = new HttpClient( connectionManager ); this.httpClient.setConnectionTimeout( 3 ); this.httpClient.setTimeout( 3 ); and then within a thread, the thread does this: String content; HttpMethod method; try { method = new PostMethod( connectionURL ); method.setDoAuthentication( true ); method.setRequestHeader( "Content-Type", contentType + "; charset=" + this.outboundEncoding); if( method instanceof PostMethod ) { PostMethod postMethod = (PostMethod) method; postMethod.setRequestBody( content ); } int responseCode = this.httpClient.executeMethod( method ); String response = method.getResponseBodyAsString(); . } catch( Exception ex ) {} finally { if( method != null ) method.releaseConnection(); } What I am doing, then, from a JUnit test class, is: 1) Send one HTTP POST to a URL, which works and I get the response. 2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30 seconds ) 3) Send another HTTP POST to the same URL. What I am seeing with ethereal is that, after 30 seconds of no activity ( no TCP ACKs whatever on the socket ), the web server sends a a TLS alert. So what actually happens is this: 1) Send one HTTP POST to a URL, which works and I get the response. 2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30 seconds ) 3) Web server sends a TLS alert after 30 seconds of inactivity. Web server sends a TCP FIN, Java sends a TCP ACK _only_ . 4) Wake up from sleep at the 40 second mark, send another HTTP POST to the same URL. 5) Receive a TLS alert instead of a HTTP response. The TLS alert according to JSSE's debug mode is: main, RECV TLSv1 ALERT: warning, close_notify main, called closeInternal(false) main, SEND TLSv1 ALERT: warning, description = close_notify 6) HttpClient throws an HttpRecoverableException, shown below: org.apache.commons.httpclient.HttpRecoverableException: org.apache.commons.httpclient.HttpRecoverableException: Error in parsing the status line from the response: unable to find line starting with "HTTP" Question is, was I correct in initially assuming that MultiThreadedHttpConnectionManager should have "handled" this case ? e.g... .detected that the exception, and retried the HTTP POST by creating a new HTTPS socket ? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] *** The information in this email is confidential and may be legally privileged. Access to this email by anyone other than the intended addressee is unauthorized. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any at
RE: Persistent HTTPS connections
John, Please correct me if I am wrong (which may well be the case) SO_TIMEOUT only affects socket read operations. I thought it had nothing to do with SSL inactivity timeout. But it looks like it might. There's another way to deal with recoverable exceptions. You can provide a custom implementation of MethodRetryHandler http://jakarta.apache.org/commons/httpclient/apidocs/org/apache/commons/httpclient/MethodRetryHandler.html The default implementation of the MethodRetryHandler is quite conservative. It does not allow the method to be retried if the request has been transmitted in its entirety. http://jakarta.apache.org/commons/httpclient/xref/org/apache/commons/httpclient/DefaultMethodRetryHandler.html#62 Oleg -Original Message- From: Jesus M. Salvo Jr. [mailto:[EMAIL PROTECTED] Sent: Thursday, May 13, 2004 7:40 To: [EMAIL PROTECTED] Subject: Persistent HTTPS connections Using HttpClient 2.0 JDK 1.4.2_04 on Fedora Is there anything special that I have to do to make use of persistent HTTP(S) connections with HttpClient other than using MultiThreadedHttpConnectionManager ? Basically, what I am doing is the following ( more explanation after the snippet of the source code ): MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager(); connectionManager.setConnectionStaleCheckingEnabled( true ); connectionManager.setMaxConnectionsPerHost( 10 ); connectionManager.setMaxTotalConnections( 100 ); this.httpClient = new HttpClient( connectionManager ); this.httpClient.setConnectionTimeout( 3 ); this.httpClient.setTimeout( 3 ); and then within a thread, the thread does this: String content; HttpMethod method; try { method = new PostMethod( connectionURL ); method.setDoAuthentication( true ); method.setRequestHeader( "Content-Type", contentType + "; charset=" + this.outboundEncoding); if( method instanceof PostMethod ) { PostMethod postMethod = (PostMethod) method; postMethod.setRequestBody( content ); } int responseCode = this.httpClient.executeMethod( method ); String response = method.getResponseBodyAsString(); . } catch( Exception ex ) {} finally { if( method != null ) method.releaseConnection(); } What I am doing, then, from a JUnit test class, is: 1) Send one HTTP POST to a URL, which works and I get the response. 2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30 seconds ) 3) Send another HTTP POST to the same URL. What I am seeing with ethereal is that, after 30 seconds of no activity ( no TCP ACKs whatever on the socket ), the web server sends a a TLS alert. So what actually happens is this: 1) Send one HTTP POST to a URL, which works and I get the response. 2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30 seconds ) 3) Web server sends a TLS alert after 30 seconds of inactivity. Web server sends a TCP FIN, Java sends a TCP ACK _only_ . 4) Wake up from sleep at the 40 second mark, send another HTTP POST to the same URL. 5) Receive a TLS alert instead of a HTTP response. The TLS alert according to JSSE's debug mode is: main, RECV TLSv1 ALERT: warning, close_notify main, called closeInternal(false) main, SEND TLSv1 ALERT: warning, description = close_notify 6) HttpClient throws an HttpRecoverableException, shown below: org.apache.commons.httpclient.HttpRecoverableException: org.apache.commons.httpclient.HttpRecoverableException: Error in parsing the status line from the response: unable to find line starting with "HTTP" Question is, was I correct in initially assuming that MultiThreadedHttpConnectionManager should have "handled" this case ? e.g... .detected that the exception, and retried the HTTP POST by creating a new HTTPS socket ? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] *** The information in this email is confidential and may be legally privileged. Access to this email by anyone other than the intended addressee is unauthorized. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system. *** - To unsubscribe, e-mail:
Re: Persistent HTTPS connections
Jesus M. Salvo Jr. wrote: Question is, was I correct in initially assuming that MultiThreadedHttpConnectionManager should have "handled" this case ? e.g... .detected that the exception, and retried the HTTP POST by creating a new HTTPS socket ? What I have done now is, if I get a HttpRecoverableException, I retry the HTTP call. The executeMethod() is therefore in a while loop. I am now to put a hard limit on the number of retries. Just like to ask what have everyone else doing. Is this the "best pratice" with HttpClient to handle persistent connections ? Regards, John - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Persistent HTTPS connections
Some clarifications below. Jesus M. Salvo Jr. wrote: What I am seeing with ethereal is that, after 30 seconds of no activity ( no TCP ACKs whatever on the socket ), the web server sends a a TLS alert. So what actually happens is this: 1) Send one HTTP POST to a URL, which works and I get the response. 2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30 seconds ) 3) Web server sends a TLS alert after 30 seconds of inactivity. Web server sends a TCP FIN, Java sends a TCP ACK _only_ . At this point, local socket is in CLOSE_WAIT state. 4) Wake up from sleep at the 40 second mark, send another HTTP POST to the same URL. 5) Receive a TLS alert instead of a HTTP response. The TLS alert according to JSSE's debug mode is: main, RECV TLSv1 ALERT: warning, close_notify main, called closeInternal(false) main, SEND TLSv1 ALERT: warning, description = close_notify Item (5) is actually not the webserver sending a TLS alert, but JSSE sending the TLS alert 6) HttpClient throws an HttpRecoverableException, shown below: org.apache.commons.httpclient.HttpRecoverableException: org.apache.commons.httpclient.HttpRecoverableException: Error in parsing the status line from the response: unable to find line starting with "HTTP" Question is, was I correct in initially assuming that MultiThreadedHttpConnectionManager should have "handled" this case ? e.g... .detected that the exception, and retried the HTTP POST by creating a new HTTPS socket ? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Persistent HTTPS connections
Using HttpClient 2.0 JDK 1.4.2_04 on Fedora Is there anything special that I have to do to make use of persistent HTTP(S) connections with HttpClient other than using MultiThreadedHttpConnectionManager ? Basically, what I am doing is the following ( more explanation after the snippet of the source code ): MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager(); connectionManager.setConnectionStaleCheckingEnabled( true ); connectionManager.setMaxConnectionsPerHost( 10 ); connectionManager.setMaxTotalConnections( 100 ); this.httpClient = new HttpClient( connectionManager ); this.httpClient.setConnectionTimeout( 3 ); this.httpClient.setTimeout( 3 ); and then within a thread, the thread does this: String content; HttpMethod method; try { method = new PostMethod( connectionURL ); method.setDoAuthentication( true ); method.setRequestHeader( "Content-Type", contentType + "; charset=" + this.outboundEncoding); if( method instanceof PostMethod ) { PostMethod postMethod = (PostMethod) method; postMethod.setRequestBody( content ); } int responseCode = this.httpClient.executeMethod( method ); String response = method.getResponseBodyAsString(); . } catch( Exception ex ) {} finally { if( method != null ) method.releaseConnection(); } What I am doing, then, from a JUnit test class, is: 1) Send one HTTP POST to a URL, which works and I get the response. 2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30 seconds ) 3) Send another HTTP POST to the same URL. What I am seeing with ethereal is that, after 30 seconds of no activity ( no TCP ACKs whatever on the socket ), the web server sends a a TLS alert. So what actually happens is this: 1) Send one HTTP POST to a URL, which works and I get the response. 2) Sleep for 40 seconds ( which is greater than the SO_TIMEOUT of 30 seconds ) 3) Web server sends a TLS alert after 30 seconds of inactivity. Web server sends a TCP FIN, Java sends a TCP ACK _only_ . 4) Wake up from sleep at the 40 second mark, send another HTTP POST to the same URL. 5) Receive a TLS alert instead of a HTTP response. The TLS alert according to JSSE's debug mode is: main, RECV TLSv1 ALERT: warning, close_notify main, called closeInternal(false) main, SEND TLSv1 ALERT: warning, description = close_notify 6) HttpClient throws an HttpRecoverableException, shown below: org.apache.commons.httpclient.HttpRecoverableException: org.apache.commons.httpclient.HttpRecoverableException: Error in parsing the status line from the response: unable to find line starting with "HTTP" Question is, was I correct in initially assuming that MultiThreadedHttpConnectionManager should have "handled" this case ? e.g... .detected that the exception, and retried the HTTP POST by creating a new HTTPS socket ? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]