Re: Help/Advise debugging SocketException in BHttpConnectionBase.fillInputBuffer()
Hi Oleg, Thanks for your quick reply. I'll try upgrading to see if that resolves this issue, On Wed, Jun 22, 2016 at 5:23 PM Oleg Kalnichevski wrote: > On Wed, 2016-06-22 at 13:00 +0200, Mark Rekveld wrote: > > Hi, > > > > I would like to ask for your assistance to resolve an issue that we are > > facing that has the nature of being a race condition. > > > > The problem is that a SocketException is thrown in the finally clause of > > BHttpConnectionBase.fillInputBuffer() [ > > > https://github.com/apache/httpcore/blob/4.3.x/httpcore/src/main/java/org/apache/http/impl/BHttpConnectionBase.java#L326 > > ] > > > > We use 4.3.6 of the HttpClient with 4.3.3 of HttpCore, and have a custom > > extension of org.apache.http.protocol.HttpRequestExecutor that sends the > > request entity asynchronously and immediately try to read the response > from > > the sever. > > > > We can only adequately support two branches of HttpClient: stable > (currently 4.5.x) and dev (currently 5.0-alpha) > > Please consider upgrading to HttpClient 4.5. There have been numerous > changes since 4.3.6. > > Oleg > > > The exception occurs within this custom HttpRequestExecutor when the > > conn.isResponseAvailable(1000 * 5) call is made in this code snippet. See > > stack trace below > > > > private void sendRequestEntity(final HttpEntityEnclosingRequest req) > throws > > IOException { > > Thread entitySender = new Thread(getClass().getSimpleName() + "-" + > > System.currentTimeMillis()) > > { > > @Override > > public void run() { > > try { > > conn.sendRequestEntity(req); > > conn.flush(); > > context.setAttribute(HttpCoreContext.HTTP_REQ_SENT, > Boolean.TRUE); > > } > > catch (Throwable t) { > > error = t; > > } > > } > > }; > > entitySender.start(); > > > > while (entitySender.isAlive() && !conn.isResponseAvailable(1000 * 5)) { > > /* > > * Wait here until either we have fully uploaded the request entity > or > > the server sends a response. If > > * we were to return immediately and called doReceiveResponse() right > > away we would provoke socket > > * timeouts. > > */ > > } > > } > > > > The full stack trace: > > > > 2016-06-03 02:36:04,149+1000 ERROR [qtp761509273-12469 - PUT > > /rest/ci/scan/treasury-web-deployable] jenkins-server > > com.sonatype.insight.brain.hds.HdsClient - Socket is closed > > java.net.SocketException: Socket is closed > > at java.net.Socket.setSoTimeout(Socket.java:1137) ~[na:1.8.0_91] > > at > > > sun.security.ssl.BaseSSLSocketImpl.setSoTimeout(BaseSSLSocketImpl.java:633) > > ~[na:1.8.0_91] > > at sun.security.ssl.SSLSocketImpl.setSoTimeout(SSLSocketImpl.java:2526) > > ~[na:1.8.0_91] > > at > > org.apache.http.impl.BHttpConnectionBase.fillInputBuffer(SourceFile:326) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.apache.http.impl.BHttpConnectionBase.awaitInput(SourceFile:334) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.http.impl.DefaultBHttpClientConnection.isResponseAvailable(SourceFile:131) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > org.apache.http.impl.conn.CPoolProxy.isResponseAvailable(SourceFile:141) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.sendRequestEntity(SourceFile:201) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.doSendRequest(SourceFile:177) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.perform(SourceFile:108) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.client.utils.HttpRequestExecutor.execute(SourceFile:53) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.apache.http.impl.execchain.MainClientExec.execute(SourceFile:254) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.apache.http.impl.execchain.ProtocolExec.execute(SourceFile:195) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.apache.http.impl.execchain.RetryExec.execute(SourceFile:86) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.apache.http.impl.execchain.RedirectExec.execute(SourceFile:108) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > org.apache.http.impl.client.InternalHttpClient.doExecute(SourceFile:184) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:82) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:106) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:57) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:380) > > [nexus-iq-server-1.20.0-02.jar:1
Re: Telling HC to make new connections all the time
On Wed, 2016-06-22 at 07:07 -0400, Benson Margulies wrote: > On Wed, Jun 22, 2016 at 3:05 AM, Stefan Magnus Landrø > wrote: > > Hi Benson, > > > > Have you considered implementing a custom > > https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/DnsResolver.html > > that handles DNS changes in a better way than the default implementation? > > You could probably use http://www.xbill.org/dnsjava/ and lookup names > > directly with the amazon NS? > > Thanks for reminding me to ask about that. Does HC call the resolver > as part of deciding whether to keep a connection or not? No, it does not, but one could make it do so by using a custom ConnectionReuseStrategy. Oleg > The problem > as we see it is this: the DNS mapping for an ELB changes, and a minute > later the connections to it become useless but don't necessarily get > properly closed with a FIN by the ELB. I've written some goofy code > that notices when the address resolution changes for any of the ELB's > we care about, and when it does it calls 'closeIdleConnections' with a > time of 0. On the other hand, I don't think we're currently having any > problems getting InetAddress.getByName to keep up with Amazon, but > your suggestion would be safer. > > > > > > Stefan > > > > 2016-06-20 19:57 GMT+02:00 Benson Margulies : > > > >> We use the async http components library to connect various components > >> in a system that is deployed, in some cases, on AWS. When it's > >> deployed on AWS, there are ELBs involved. > >> > >> Like some other people, we've having various trauma dealing with the > >> very dynamic environment of AWS ELBs, which includes constant DNS > >> changes and some other issues. So far, we've been making our HC > >> configuration more and more complex. I'm wondering if we should back > >> off and make it simpler -- make a new connection for each request. > >> Obviously, this will be somewhat slower and more expensive, but at the > >> moment we might prefer a slower and more expensive approach that works > >> to one that occasionally gets lost. > >> > >> I appreciate that even making new connections all the time won't > >> compensate for all possible DNS problems -- if Java itself caches a > >> stale name->address translation, making a new connection will fail > >> just as effectively as anything else. > >> > >> Does anyone out there have advice to share (other than running away from > >> ELB)? > >> > >> - > >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org > >> > >> > > > > > > -- > > BEKK Open > > http://open.bekk.no > > > > TesTcl - a unit test framework for iRules > > http://testcl.com > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Help/Advise debugging SocketException in BHttpConnectionBase.fillInputBuffer()
On Wed, 2016-06-22 at 13:00 +0200, Mark Rekveld wrote: > Hi, > > I would like to ask for your assistance to resolve an issue that we are > facing that has the nature of being a race condition. > > The problem is that a SocketException is thrown in the finally clause of > BHttpConnectionBase.fillInputBuffer() [ > https://github.com/apache/httpcore/blob/4.3.x/httpcore/src/main/java/org/apache/http/impl/BHttpConnectionBase.java#L326 > ] > > We use 4.3.6 of the HttpClient with 4.3.3 of HttpCore, and have a custom > extension of org.apache.http.protocol.HttpRequestExecutor that sends the > request entity asynchronously and immediately try to read the response from > the sever. > We can only adequately support two branches of HttpClient: stable (currently 4.5.x) and dev (currently 5.0-alpha) Please consider upgrading to HttpClient 4.5. There have been numerous changes since 4.3.6. Oleg > The exception occurs within this custom HttpRequestExecutor when the > conn.isResponseAvailable(1000 * 5) call is made in this code snippet. See > stack trace below > > private void sendRequestEntity(final HttpEntityEnclosingRequest req) throws > IOException { > Thread entitySender = new Thread(getClass().getSimpleName() + "-" + > System.currentTimeMillis()) > { > @Override > public void run() { > try { > conn.sendRequestEntity(req); > conn.flush(); > context.setAttribute(HttpCoreContext.HTTP_REQ_SENT, Boolean.TRUE); > } > catch (Throwable t) { > error = t; > } > } > }; > entitySender.start(); > > while (entitySender.isAlive() && !conn.isResponseAvailable(1000 * 5)) { > /* > * Wait here until either we have fully uploaded the request entity or > the server sends a response. If > * we were to return immediately and called doReceiveResponse() right > away we would provoke socket > * timeouts. > */ > } > } > > The full stack trace: > > 2016-06-03 02:36:04,149+1000 ERROR [qtp761509273-12469 - PUT > /rest/ci/scan/treasury-web-deployable] jenkins-server > com.sonatype.insight.brain.hds.HdsClient - Socket is closed > java.net.SocketException: Socket is closed > at java.net.Socket.setSoTimeout(Socket.java:1137) ~[na:1.8.0_91] > at > sun.security.ssl.BaseSSLSocketImpl.setSoTimeout(BaseSSLSocketImpl.java:633) > ~[na:1.8.0_91] > at sun.security.ssl.SSLSocketImpl.setSoTimeout(SSLSocketImpl.java:2526) > ~[na:1.8.0_91] > at > org.apache.http.impl.BHttpConnectionBase.fillInputBuffer(SourceFile:326) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at org.apache.http.impl.BHttpConnectionBase.awaitInput(SourceFile:334) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at > org.apache.http.impl.DefaultBHttpClientConnection.isResponseAvailable(SourceFile:131) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at > org.apache.http.impl.conn.CPoolProxy.isResponseAvailable(SourceFile:141) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at > com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.sendRequestEntity(SourceFile:201) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at > com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.doSendRequest(SourceFile:177) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at > com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.perform(SourceFile:108) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at > com.sonatype.insight.client.utils.HttpRequestExecutor.execute(SourceFile:53) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at org.apache.http.impl.execchain.MainClientExec.execute(SourceFile:254) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at org.apache.http.impl.execchain.ProtocolExec.execute(SourceFile:195) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at org.apache.http.impl.execchain.RetryExec.execute(SourceFile:86) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at org.apache.http.impl.execchain.RedirectExec.execute(SourceFile:108) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at > org.apache.http.impl.client.InternalHttpClient.doExecute(SourceFile:184) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:82) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:106) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:57) > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:380) > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:329) > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at com.sonatype.insight.brain.hds.HdsClient.getResponse(SourceFile:132) > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at com.sonatype.insight.brain.hds.HdsClient.get(SourceFile:178) > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > at com.sonatype.insight.brain.hds.Sc
Re: Telling HC to make new connections all the time
On Wed, Jun 22, 2016 at 3:05 AM, Stefan Magnus Landrø wrote: > Hi Benson, > > Have you considered implementing a custom > https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/DnsResolver.html > that handles DNS changes in a better way than the default implementation? > You could probably use http://www.xbill.org/dnsjava/ and lookup names > directly with the amazon NS? Thanks for reminding me to ask about that. Does HC call the resolver as part of deciding whether to keep a connection or not? The problem as we see it is this: the DNS mapping for an ELB changes, and a minute later the connections to it become useless but don't necessarily get properly closed with a FIN by the ELB. I've written some goofy code that notices when the address resolution changes for any of the ELB's we care about, and when it does it calls 'closeIdleConnections' with a time of 0. On the other hand, I don't think we're currently having any problems getting InetAddress.getByName to keep up with Amazon, but your suggestion would be safer. > > Stefan > > 2016-06-20 19:57 GMT+02:00 Benson Margulies : > >> We use the async http components library to connect various components >> in a system that is deployed, in some cases, on AWS. When it's >> deployed on AWS, there are ELBs involved. >> >> Like some other people, we've having various trauma dealing with the >> very dynamic environment of AWS ELBs, which includes constant DNS >> changes and some other issues. So far, we've been making our HC >> configuration more and more complex. I'm wondering if we should back >> off and make it simpler -- make a new connection for each request. >> Obviously, this will be somewhat slower and more expensive, but at the >> moment we might prefer a slower and more expensive approach that works >> to one that occasionally gets lost. >> >> I appreciate that even making new connections all the time won't >> compensate for all possible DNS problems -- if Java itself caches a >> stale name->address translation, making a new connection will fail >> just as effectively as anything else. >> >> Does anyone out there have advice to share (other than running away from >> ELB)? >> >> - >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org >> >> > > > -- > BEKK Open > http://open.bekk.no > > TesTcl - a unit test framework for iRules > http://testcl.com - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Help/Advise debugging SocketException in BHttpConnectionBase.fillInputBuffer()
Hi, I would like to ask for your assistance to resolve an issue that we are facing that has the nature of being a race condition. The problem is that a SocketException is thrown in the finally clause of BHttpConnectionBase.fillInputBuffer() [ https://github.com/apache/httpcore/blob/4.3.x/httpcore/src/main/java/org/apache/http/impl/BHttpConnectionBase.java#L326 ] We use 4.3.6 of the HttpClient with 4.3.3 of HttpCore, and have a custom extension of org.apache.http.protocol.HttpRequestExecutor that sends the request entity asynchronously and immediately try to read the response from the sever. The exception occurs within this custom HttpRequestExecutor when the conn.isResponseAvailable(1000 * 5) call is made in this code snippet. See stack trace below private void sendRequestEntity(final HttpEntityEnclosingRequest req) throws IOException { Thread entitySender = new Thread(getClass().getSimpleName() + "-" + System.currentTimeMillis()) { @Override public void run() { try { conn.sendRequestEntity(req); conn.flush(); context.setAttribute(HttpCoreContext.HTTP_REQ_SENT, Boolean.TRUE); } catch (Throwable t) { error = t; } } }; entitySender.start(); while (entitySender.isAlive() && !conn.isResponseAvailable(1000 * 5)) { /* * Wait here until either we have fully uploaded the request entity or the server sends a response. If * we were to return immediately and called doReceiveResponse() right away we would provoke socket * timeouts. */ } } The full stack trace: 2016-06-03 02:36:04,149+1000 ERROR [qtp761509273-12469 - PUT /rest/ci/scan/treasury-web-deployable] jenkins-server com.sonatype.insight.brain.hds.HdsClient - Socket is closed java.net.SocketException: Socket is closed at java.net.Socket.setSoTimeout(Socket.java:1137) ~[na:1.8.0_91] at sun.security.ssl.BaseSSLSocketImpl.setSoTimeout(BaseSSLSocketImpl.java:633) ~[na:1.8.0_91] at sun.security.ssl.SSLSocketImpl.setSoTimeout(SSLSocketImpl.java:2526) ~[na:1.8.0_91] at org.apache.http.impl.BHttpConnectionBase.fillInputBuffer(SourceFile:326) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at org.apache.http.impl.BHttpConnectionBase.awaitInput(SourceFile:334) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at org.apache.http.impl.DefaultBHttpClientConnection.isResponseAvailable(SourceFile:131) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at org.apache.http.impl.conn.CPoolProxy.isResponseAvailable(SourceFile:141) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.sendRequestEntity(SourceFile:201) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.doSendRequest(SourceFile:177) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.perform(SourceFile:108) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.sonatype.insight.client.utils.HttpRequestExecutor.execute(SourceFile:53) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at org.apache.http.impl.execchain.MainClientExec.execute(SourceFile:254) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at org.apache.http.impl.execchain.ProtocolExec.execute(SourceFile:195) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at org.apache.http.impl.execchain.RetryExec.execute(SourceFile:86) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at org.apache.http.impl.execchain.RedirectExec.execute(SourceFile:108) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at org.apache.http.impl.client.InternalHttpClient.doExecute(SourceFile:184) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:82) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:106) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:57) ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:380) [nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:329) [nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.sonatype.insight.brain.hds.HdsClient.getResponse(SourceFile:132) [nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.sonatype.insight.brain.hds.HdsClient.get(SourceFile:178) [nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.sonatype.insight.brain.hds.ScanUploader.upload(SourceFile:69) [nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.sonatype.insight.brain.hds.CIResource.putScan(SourceFile:50) [nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8.CGLIB$putScan$0() [nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8$$FastClassByGuice$$99131cf1.invoke() [nexus-iq-server-1.20.0-02.jar:1.20.0-02] at com.google.injec
Re: Telling HC to make new connections all the time
Hi Benson, Have you considered implementing a custom https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/DnsResolver.html that handles DNS changes in a better way than the default implementation? You could probably use http://www.xbill.org/dnsjava/ and lookup names directly with the amazon NS? Stefan 2016-06-20 19:57 GMT+02:00 Benson Margulies : > We use the async http components library to connect various components > in a system that is deployed, in some cases, on AWS. When it's > deployed on AWS, there are ELBs involved. > > Like some other people, we've having various trauma dealing with the > very dynamic environment of AWS ELBs, which includes constant DNS > changes and some other issues. So far, we've been making our HC > configuration more and more complex. I'm wondering if we should back > off and make it simpler -- make a new connection for each request. > Obviously, this will be somewhat slower and more expensive, but at the > moment we might prefer a slower and more expensive approach that works > to one that occasionally gets lost. > > I appreciate that even making new connections all the time won't > compensate for all possible DNS problems -- if Java itself caches a > stale name->address translation, making a new connection will fail > just as effectively as anything else. > > Does anyone out there have advice to share (other than running away from > ELB)? > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > -- BEKK Open http://open.bekk.no TesTcl - a unit test framework for iRules http://testcl.com