We have an odd issue wherein at Tomcat(TomEE) startup apache-hc execution
requests throw a NoHttpResponseException within less than 1 second. After a
few minutes this behavior completely stops and isn’t seen again until the next
time we stop and restart Tomcat(TomEE).
Basic apache-hc configuration information:
- single static CloseableHttpClient instance initialized at container startup;
- all requests managed via PoolingHttpClientConnectionManager
- connection timeout 6000ms
- read timeout 6000ms
- we use 4 retries
- the typical timing for each HttpClient.execute(HttpUriRequest,HttpContext)
during this startup period is that they fail very fast:
try-1: ~ 5ms
try-2: ~250ms
try-3: ~500ms
try-5: ~700ms
We measure the elapsed milli-seconds with respect to “execute()”. It seems
very odd that execute() fails so fast given the apache-hc settings. It isn’t
every execution request that fails. But, a very high percentage of them do
exhibit this behavior during the initial ~5 minutes after Tomcat(TomEE) begins
dispatching inbound requests.
The inbound HTTP request volume inbound is ~70 RPS(requests per second).
After ~5 minutes this type of error never repeats. We have logging in place
that verifies all the apache-hc components are fully instantiated prior to any
request being processed.
Version information:
apache-hc: 4.5.2
java : 1.8r60
tomcat : 7.0.68
tomee : 1.7.4
Any ideas at all given the minimal information that I have to provide are
greatly appreciated.
—
Pete
