Hi Oleg, Thanks for your quick reply. I'll try upgrading to see if that resolves this issue,
On Wed, Jun 22, 2016 at 5:23 PM Oleg Kalnichevski <ol...@apache.org> wrote: > On Wed, 2016-06-22 at 13:00 +0200, Mark Rekveld wrote: > > Hi, > > > > I would like to ask for your assistance to resolve an issue that we are > > facing that has the nature of being a race condition. > > > > The problem is that a SocketException is thrown in the finally clause of > > BHttpConnectionBase.fillInputBuffer() [ > > > https://github.com/apache/httpcore/blob/4.3.x/httpcore/src/main/java/org/apache/http/impl/BHttpConnectionBase.java#L326 > > ] > > > > We use 4.3.6 of the HttpClient with 4.3.3 of HttpCore, and have a custom > > extension of org.apache.http.protocol.HttpRequestExecutor that sends the > > request entity asynchronously and immediately try to read the response > from > > the sever. > > > > We can only adequately support two branches of HttpClient: stable > (currently 4.5.x) and dev (currently 5.0-alpha) > > Please consider upgrading to HttpClient 4.5. There have been numerous > changes since 4.3.6. > > Oleg > > > The exception occurs within this custom HttpRequestExecutor when the > > conn.isResponseAvailable(1000 * 5) call is made in this code snippet. See > > stack trace below > > > > private void sendRequestEntity(final HttpEntityEnclosingRequest req) > throws > > IOException { > > Thread entitySender = new Thread(getClass().getSimpleName() + "-" + > > System.currentTimeMillis()) > > { > > @Override > > public void run() { > > try { > > conn.sendRequestEntity(req); > > conn.flush(); > > context.setAttribute(HttpCoreContext.HTTP_REQ_SENT, > Boolean.TRUE); > > } > > catch (Throwable t) { > > error = t; > > } > > } > > }; > > entitySender.start(); > > > > while (entitySender.isAlive() && !conn.isResponseAvailable(1000 * 5)) { > > /* > > * Wait here until either we have fully uploaded the request entity > or > > the server sends a response. If > > * we were to return immediately and called doReceiveResponse() right > > away we would provoke socket > > * timeouts. > > */ > > } > > } > > > > The full stack trace: > > > > 2016-06-03 02:36:04,149+1000 ERROR [qtp761509273-12469 - PUT > > /rest/ci/scan/treasury-web-deployable] jenkins-server > > com.sonatype.insight.brain.hds.HdsClient - Socket is closed > > java.net.SocketException: Socket is closed > > at java.net.Socket.setSoTimeout(Socket.java:1137) ~[na:1.8.0_91] > > at > > > sun.security.ssl.BaseSSLSocketImpl.setSoTimeout(BaseSSLSocketImpl.java:633) > > ~[na:1.8.0_91] > > at sun.security.ssl.SSLSocketImpl.setSoTimeout(SSLSocketImpl.java:2526) > > ~[na:1.8.0_91] > > at > > org.apache.http.impl.BHttpConnectionBase.fillInputBuffer(SourceFile:326) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.apache.http.impl.BHttpConnectionBase.awaitInput(SourceFile:334) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.http.impl.DefaultBHttpClientConnection.isResponseAvailable(SourceFile:131) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > org.apache.http.impl.conn.CPoolProxy.isResponseAvailable(SourceFile:141) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.sendRequestEntity(SourceFile:201) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.doSendRequest(SourceFile:177) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.perform(SourceFile:108) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.client.utils.HttpRequestExecutor.execute(SourceFile:53) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.apache.http.impl.execchain.MainClientExec.execute(SourceFile:254) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.apache.http.impl.execchain.ProtocolExec.execute(SourceFile:195) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.apache.http.impl.execchain.RetryExec.execute(SourceFile:86) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.apache.http.impl.execchain.RedirectExec.execute(SourceFile:108) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > org.apache.http.impl.client.InternalHttpClient.doExecute(SourceFile:184) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:82) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:106) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:57) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:380) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:329) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.sonatype.insight.brain.hds.HdsClient.getResponse(SourceFile:132) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.sonatype.insight.brain.hds.HdsClient.get(SourceFile:178) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.sonatype.insight.brain.hds.ScanUploader.upload(SourceFile:69) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.sonatype.insight.brain.hds.CIResource.putScan(SourceFile:50) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8.CGLIB$putScan$0(<generated>) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8$$FastClassByGuice$$99131cf1.invoke(<generated>) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(SourceFile:228) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(SourceFile:75) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(SourceFile:49) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.brain.security.AuthorizeMethodInterceptor.invoke(SourceFile:62) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(SourceFile:36) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(SourceFile:75) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.google.inject.internal.InterceptorStackCallback.intercept(SourceFile:55) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8.putScan(<generated>) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source) > ~[na:na] > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > ~[na:1.8.0_91] > > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91] > > at > > > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(SourceFile:60) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(SourceFile:185) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(SourceFile:75) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.yammer.dropwizard.jersey.OptionalResourceMethodDispatchAdapter$OptionalRequestDispatcher.dispatch(SourceFile:37) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(SourceFile:302) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(SourceFile:147) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(SourceFile:108) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(SourceFile:147) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(SourceFile:84) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(SourceFile:1511) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(SourceFile:1442) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(SourceFile:1391) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(SourceFile:1381) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > com.sun.jersey.spi.container.servlet.WebComponent.service(SourceFile:416) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(SourceFile:538) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(SourceFile:716) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at javax.servlet.http.HttpServlet.service(SourceFile:848) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.servlet.ServletHolder.handle(SourceFile:669) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1448) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.brain.security.AuthenticationLoggingFilter.doFilter(SourceFile:53) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:61) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.shiro.web.servlet.AdviceFilter.executeChain(SourceFile:108) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(SourceFile:137) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:118) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.shiro.web.servlet.AdviceFilter.executeChain(SourceFile:108) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(SourceFile:137) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.shiro.web.servlet.AdviceFilter.executeChain(SourceFile:108) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(SourceFile:137) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(SourceFile:449) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(SourceFile:365) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.apache.shiro.subject.support.SubjectCallable.doCall(SourceFile:90) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.apache.shiro.subject.support.SubjectCallable.call(SourceFile:83) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.subject.support.DelegatingSubject.execute(SourceFile:383) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(SourceFile:362) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.brain.security.TraceMethodBlockFilter.doFilter(SourceFile:48) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > com.yammer.dropwizard.servlets.ThreadNameFilter.doFilter(SourceFile:29) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.servlet.ServletHandler.doHandle(SourceFile:455) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(SourceFile:1075) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.servlet.ServletHandler.doScope(SourceFile:384) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > org.eclipse.jetty.server.handler.ContextHandler.doScope(SourceFile:1009) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.server.handler.ScopedHandler.handle(SourceFile:135) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(SourceFile:116) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.yammer.metrics.jetty.InstrumentedHandler.handle(SourceFile:200) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.server.handler.GzipHandler.handle(SourceFile:264) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.yammer.dropwizard.jetty.BiDiGzipHandler.handle(SourceFile:123) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > org.eclipse.jetty.server.handler.HandlerCollection.handle(SourceFile:154) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(SourceFile:116) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.server.Server.handle(SourceFile:368) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(SourceFile:489) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > org.eclipse.jetty.server.AbstractHttpConnection.content(SourceFile:953) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(SourceFile:1014) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.http.HttpParser.parseNext(SourceFile:861) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.http.HttpParser.parseAvailable(SourceFile:240) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.server.AsyncHttpConnection.handle(SourceFile:82) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SourceFile:628) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SourceFile:52) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(SourceFile:608) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(SourceFile:543) > > [nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] > > 2016-06-03 02:36:04,149+1000 DEBUG [qtp761509273-12469 - PUT > > /rest/ci/scan/treasury-web-deployable] jenkins-server > > com.sonatype.insight.brain.hds.HdsClient - Completed HDS request in 9381 > ms. > > 2016-06-03 02:36:04,151+1000 ERROR [qtp761509273-12469 - PUT > > /rest/ci/scan/treasury-web-deployable] jenkins-server > > com.sonatype.insight.jaxrs.error.ErrorResponseGenerator - The request to > > Sonatype HDS failed, please retry in a bit. > > com.sonatype.insight.error.exception.BadGatewayException: The request to > > Sonatype HDS failed, please retry in a bit. > > at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:391) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:329) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.sonatype.insight.brain.hds.HdsClient.getResponse(SourceFile:132) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.sonatype.insight.brain.hds.HdsClient.get(SourceFile:178) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.sonatype.insight.brain.hds.ScanUploader.upload(SourceFile:69) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at com.sonatype.insight.brain.hds.CIResource.putScan(SourceFile:50) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(SourceFile:49) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > com.sonatype.insight.brain.security.AuthorizeMethodInterceptor.invoke(SourceFile:62) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > at > > > org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(SourceFile:36) > > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > -- Cheers, Mark Rekveld
