Oleg, Thanks very much for the reply. I mentioned the SO_TIMEOUT setting just to explain the condition that led to the exception and seems to have ended up leaving the socket in a bad state. I understand too that CLOSE_WAIT is due to the server having closed the socket but httpclient has not. We do run the idle connection timeout thread, and unfortunately in this case it doesn't seem to help.
The focus of my attention has been the HttpConnection.closeSocketAndStreams() method which is where the socket seems to end up being disposed of both by the IdleConnectionTimeoutThread and all other logic that closes the connection. In 3.0.1 and 3.1, this simply calls close() on the streams and sockets. In 4.x however, the SocketHttpClientConnection.close() method does perform a shutdownOutput()/shutdownInput(). Some snippets of information I've found seem to indicate that on some socket stacks close() is not enough to actually close a socket and make the CLOSE_WAIT->LAST_ACK transition, and that a shutdown is necessary. This seems to jive with our experience here. I was assuming that the change in behavior from 3.x to 4.x was the result of lessons learned and that maybe this had a bearing on our scenario. If not, we can run some experiments and try to isolate it further. Eventually of course we would like to migrate to the newer, cleaner, APIs. But in the meantime if anything occurs to you or anyone else about this condition, please let me know. Thanks again. Gerry -----Original Message----- From: Oleg Kalnichevski [mailto:[email protected]] Sent: Thursday, July 30, 2009 3:08 AM To: HttpClient User Discussion Subject: Re: CLOSE_WAIT on Linux On Wed, Jul 29, 2009 at 09:36:20PM -0700, Gerry Woods wrote: > I haven't posted here before so let me start by thanking you guys for the > great work. > > We are using 3.0.1 in some pretty heavy production environments. A recent > issue arose with a customer who had trouble during SSL handshakes. The > negotiation was timing out: > > [120009] java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(Unknown Source) > at com.sun.net.ssl.internal.ssl.InputRecord.a(Unknown Source) > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown > Source) > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(Unknown > Source) > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.j(Unknown > Source) > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.b(Unknown > Source) > at > com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source) > ... > > We're setting SO_TIMEOUT to 2 minutes. The customer reports that the machine > that initiated the connection ended up with a lot of sockets in CLOSE_WAIT. > From what I've been able to glean from some quick searches, it sounds like on > some socket stacks, a shutdownOutput is required, and that close() is not > enough to nudge the socket from CLOSE_WAIT to LAST_ACK. I notice that in the > 4.x codebase, shutdownOutput is being used, but in the 3.x code it is not. > Can anyone confirm that this is indeed the case for Linux? If so, is there a > known workaround other than upgrading to 4.x? > > Thanks for any help you can offer, and thanks again for all the countless > hours (and dollars!) your efforts have saved us. > > Gerry, As far as I know SO_TIMEOUT parameter and CLOSE_WAIT socket state are completely unrelated. Usually sockets end up in the CLOSE_WAIT state when a persistent connection is closed by the remote server while still kept alive in the connection pool on the client. This is general problem with the blocking I/O model and both HttpClient 3.x and 4.0 are affected by it. However, HttpClient 4.0 is much better equipped to deal with the issue. In both cases one needs to implement some sort of connection eviction policy to make sure that persistent connections that have been idle for too long get closed and evicted from the pool. http://hc.apache.org/httpcomponents-client/tutorial/html/ch02.html#d4e638 Hope this helps Oleg --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
