Thanks Oleg.
The keyword “reset” is enough to help us point networking in a plausible
direction.
- since this is a symptom associated with a broker connection this
confirms why each retry always fails
within the same 5ms elapsed time;
- we currently allow a retry using the same execution context;
- obviously a fruitless waste of resource time;
—
Pete
On 5/6/16, 5:24 AM, "Oleg Kalnichevski" <[email protected]> wrote:
>On Thu, 2016-05-05 at 14:46 +0000, Pete Keyes wrote:
>> Update for the elapsed time for these calls that fail strictly at
>>startup.
>> The stair-step 5/25/500... was strictly due to a pause we insert
>>between
>> retries. All 4 tries fail within 5ms.
>>
>> Given the configuration settings for CloseableHttpClient a failure
>>within
>> 5ms seems unlikely. The trip to the F5 VIP and back would barely
>>complete
>> within 5ms.
>>
>>
>> Any thoughts on how to stabilize this a bit at startup? Or glean more
>> information about why apache-hc would fail with NoHttpResponseException
>>so
>> quickly.
>
>The best course of action would be to capture IP packets with a traffic
>analyzer such as Wireshark and to try to figure out why TCP connections
>get reset.
>
>Oleg
>
>
>
>> Stack trace for the error:
>> org.apache.http.NoHttpResponseException:
>> crm-asp-prod.starbucks.net:443 failed to respond
>>
>>
>>org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttp
>>Re
>> sponseParser.java:143)
>>
>>
>>org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttp
>>Re
>> sponseParser.java:57)
>>
>>
>>org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser
>>.j
>> ava:261)
>>
>>
>>org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(D
>>ef
>> aultBHttpClientConnection.java:165)
>>
>>
>>org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.jav
>>a:
>> 167)
>>
>>
>>org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpReques
>>tE
>> xecutor.java:272)
>>
>>
>>org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.
>>ja
>> va:124)
>>
>>
>>org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java
>>:2
>> 71)
>>
>>
>>org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184
>>)
>> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
>>
>>
>>org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110
>>)
>>
>>
>>org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClie
>>nt
>> .java:184)
>>
>>
>>org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClie
>>nt
>> .java:82)
>>
>>
>>
>> ‹
>> Pete Keyes
>>
>>
>>
>>
>>
>>
>> On 5/4/16, 11:13 AM, "Pete Keyes" <[email protected]> wrote:
>>
>> >We have an odd issue wherein at Tomcat(TomEE) startup apache-hc
>>execution
>> >requests throw a NoHttpResponseException within less than 1 second.
>> >After a few minutes this behavior completely stops and isn¹t seen again
>> >until the next time we stop and restart Tomcat(TomEE).
>> >
>> >Basic apache-hc configuration information:
>> > - single static CloseableHttpClient instance initialized at container
>> >startup;
>> > - all requests managed via PoolingHttpClientConnectionManager
>> > - connection timeout 6000ms
>> > - read timeout 6000ms
>> > - we use 4 retries
>> > - the typical timing for each
>> >HttpClient.execute(HttpUriRequest,HttpContext) during this startup
>>period
>> >is that they fail very fast:
>> > try-1: ~ 5ms
>> > try-2: ~250ms
>> > try-3: ~500ms
>> > try-5: ~700ms
>> >We measure the elapsed milli-seconds with respect to ³execute()². It
>> >seems very odd that execute() fails so fast given the apache-hc
>>settings.
>> > It isn¹t every execution request that fails. But, a very high
>> >percentage of them do exhibit this behavior during the initial ~5
>>minutes
>> >after Tomcat(TomEE) begins dispatching inbound requests.
>> >
>> >The inbound HTTP request volume inbound is ~70 RPS(requests per
>>second).
>> >
>> >After ~5 minutes this type of error never repeats. We have logging in
>> >place that verifies all the apache-hc components are fully instantiated
>> >prior to any request being processed.
>> >
>> >Version information:
>> > apache-hc: 4.5.2
>> > java : 1.8r60
>> > tomcat : 7.0.68
>> > tomee : 1.7.4
>> >
>> >
>> >Any ideas at all given the minimal information that I have to provide
>>are
>> >greatly appreciated.
>> >
>> >‹
>> >Pete
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: [email protected]
>For additional commands, e-mail: [email protected]
>
