Re: NoHttpResponseException received at startup
On Thu, 2016-05-05 at 14:46 +, Pete Keyes wrote: > Update for the elapsed time for these calls that fail strictly at startup. > The stair-step 5/25/500... was strictly due to a pause we insert between > retries. All 4 tries fail within 5ms. > > Given the configuration settings for CloseableHttpClient a failure within > 5ms seems unlikely. The trip to the F5 VIP and back would barely complete > within 5ms. > > > Any thoughts on how to stabilize this a bit at startup? Or glean more > information about why apache-hc would fail with NoHttpResponseException so > quickly. The best course of action would be to capture IP packets with a traffic analyzer such as Wireshark and to try to figure out why TCP connections get reset. Oleg > Stack trace for the error: > org.apache.http.NoHttpResponseException: > crm-asp-prod.starbucks.net:443 failed to respond > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRe > sponseParser.java:143) > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRe > sponseParser.java:57) > > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.j > ava:261) > > org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(Def > aultBHttpClientConnection.java:165) > > org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java: > 167) > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestE > xecutor.java:272) > > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.ja > va:124) > > org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:2 > 71) > > org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) > org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) > > org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) > > org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient > .java:184) > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient > .java:82) > > > > ‹ > Pete Keyes > > > > > > > On 5/4/16, 11:13 AM, "Pete Keyes"wrote: > > >We have an odd issue wherein at Tomcat(TomEE) startup apache-hc execution > >requests throw a NoHttpResponseException within less than 1 second. > >After a few minutes this behavior completely stops and isn¹t seen again > >until the next time we stop and restart Tomcat(TomEE). > > > >Basic apache-hc configuration information: > > - single static CloseableHttpClient instance initialized at container > >startup; > > - all requests managed via PoolingHttpClientConnectionManager > > - connection timeout 6000ms > > - read timeout 6000ms > > - we use 4 retries > > - the typical timing for each > >HttpClient.execute(HttpUriRequest,HttpContext) during this startup period > >is that they fail very fast: > > try-1: ~ 5ms > > try-2: ~250ms > > try-3: ~500ms > > try-5: ~700ms > >We measure the elapsed milli-seconds with respect to ³execute()². It > >seems very odd that execute() fails so fast given the apache-hc settings. > > It isn¹t every execution request that fails. But, a very high > >percentage of them do exhibit this behavior during the initial ~5 minutes > >after Tomcat(TomEE) begins dispatching inbound requests. > > > >The inbound HTTP request volume inbound is ~70 RPS(requests per second). > > > >After ~5 minutes this type of error never repeats. We have logging in > >place that verifies all the apache-hc components are fully instantiated > >prior to any request being processed. > > > >Version information: > > apache-hc: 4.5.2 > > java : 1.8r60 > > tomcat : 7.0.68 > > tomee: 1.7.4 > > > > > >Any ideas at all given the minimal information that I have to provide are > >greatly appreciated. > > > >‹ > >Pete > > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: NoHttpResponseException received at startup
Update for the elapsed time for these calls that fail strictly at startup. The stair-step 5/25/500... was strictly due to a pause we insert between retries. All 4 tries fail within 5ms. Given the configuration settings for CloseableHttpClient a failure within 5ms seems unlikely. The trip to the F5 VIP and back would barely complete within 5ms. Any thoughts on how to stabilize this a bit at startup? Or glean more information about why apache-hc would fail with NoHttpResponseException so quickly. Stack trace for the error: org.apache.http.NoHttpResponseException: crm-asp-prod.starbucks.net:443 failed to respond org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRe sponseParser.java:143) org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRe sponseParser.java:57) org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.j ava:261) org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(Def aultBHttpClientConnection.java:165) org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java: 167) org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestE xecutor.java:272) org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.ja va:124) org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:2 71) org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient .java:184) org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient .java:82) ‹ Pete Keyes On 5/4/16, 11:13 AM, "Pete Keyes"wrote: >We have an odd issue wherein at Tomcat(TomEE) startup apache-hc execution >requests throw a NoHttpResponseException within less than 1 second. >After a few minutes this behavior completely stops and isn¹t seen again >until the next time we stop and restart Tomcat(TomEE). > >Basic apache-hc configuration information: > - single static CloseableHttpClient instance initialized at container >startup; > - all requests managed via PoolingHttpClientConnectionManager > - connection timeout 6000ms > - read timeout 6000ms > - we use 4 retries > - the typical timing for each >HttpClient.execute(HttpUriRequest,HttpContext) during this startup period >is that they fail very fast: > try-1: ~ 5ms > try-2: ~250ms > try-3: ~500ms > try-5: ~700ms >We measure the elapsed milli-seconds with respect to ³execute()². It >seems very odd that execute() fails so fast given the apache-hc settings. > It isn¹t every execution request that fails. But, a very high >percentage of them do exhibit this behavior during the initial ~5 minutes >after Tomcat(TomEE) begins dispatching inbound requests. > >The inbound HTTP request volume inbound is ~70 RPS(requests per second). > >After ~5 minutes this type of error never repeats. We have logging in >place that verifies all the apache-hc components are fully instantiated >prior to any request being processed. > >Version information: > apache-hc: 4.5.2 > java : 1.8r60 > tomcat : 7.0.68 > tomee: 1.7.4 > > >Any ideas at all given the minimal information that I have to provide are >greatly appreciated. > >‹ >Pete > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
NoHttpResponseException received at startup
We have an odd issue wherein at Tomcat(TomEE) startup apache-hc execution requests throw a NoHttpResponseException within less than 1 second. After a few minutes this behavior completely stops and isn’t seen again until the next time we stop and restart Tomcat(TomEE). Basic apache-hc configuration information: - single static CloseableHttpClient instance initialized at container startup; - all requests managed via PoolingHttpClientConnectionManager - connection timeout 6000ms - read timeout 6000ms - we use 4 retries - the typical timing for each HttpClient.execute(HttpUriRequest,HttpContext) during this startup period is that they fail very fast: try-1: ~ 5ms try-2: ~250ms try-3: ~500ms try-5: ~700ms We measure the elapsed milli-seconds with respect to “execute()”. It seems very odd that execute() fails so fast given the apache-hc settings. It isn’t every execution request that fails. But, a very high percentage of them do exhibit this behavior during the initial ~5 minutes after Tomcat(TomEE) begins dispatching inbound requests. The inbound HTTP request volume inbound is ~70 RPS(requests per second). After ~5 minutes this type of error never repeats. We have logging in place that verifies all the apache-hc components are fully instantiated prior to any request being processed. Version information: apache-hc: 4.5.2 java : 1.8r60 tomcat : 7.0.68 tomee: 1.7.4 Any ideas at all given the minimal information that I have to provide are greatly appreciated. — Pete