Oh, I actually *do* have more: Internal Failure java.lang.IllegalStateException: Endpoint not acquired / already released at com.amazon.coral.apache.hc.client5.http.impl.classic.InternalExecRuntime.ensureValid(InternalExecRuntime.java:142)
at com.amazon.coral.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172) at com.amazon.coral.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:135) at com.amazon.coral.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at com.amazon.coral.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) at com.amazon.coral.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:172) at com.amazon.coral.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at com.amazon.coral.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) at com.amazon.coral.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:93) at com.amazon.coral.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at com.amazon.coral.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:178) at com.amazon.coral.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:75) On Tue, Mar 23, 2021 at 3:11 PM Ryan Schmitt <[email protected]> wrote: > This is all I have right now, but I can ask the service owner for more if > necessary: > > ... 30 more | Caused by: java.lang.IllegalStateException: Endpoint not > acquired / already released | > at > com.amazon.coral.apache.hc.client5.http.impl.classic.InternalExecRuntime.ensureValid(InternalExecRuntime.java:142) > | > at > com.amazon.coral.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172) > | > > On Sat, Mar 20, 2021 at 2:30 AM Oleg Kalnichevski <[email protected]> > wrote: > >> On Fri, 2021-03-19 at 11:26 -0700, Ryan Schmitt wrote: >> > After rolling out our Apache 4 -> 5 upgrade again (this time, with >> > idle >> > connection validation configured), we got the following report from >> > someone >> > who discovered an issue in load testing. The service he is testing >> > has a >> > dependency that is called with high fanout (up to 30 times per >> > request), >> > with requests made in parallel from a thread pool. During load >> > testing, the >> > service totally locked up; even after stopping traffic for two hours, >> > all >> > requests to the service were failing. >> > >> > Two different types of exceptions were observed: >> > >> > 1. `CancellationException` on `BasicFuture.get()` when trying to >> > lease a >> > connection from `PoolingHttpClientConnectionManager` >> > 2. `java.lang.IllegalStateException: Endpoint not acquired / already >> > released` at >> > `com.amazon.coral.apache.hc.client5.http.impl.classic.InternalExecRun >> > time.ensureValid(InternalExecRuntime.java:142)` >> > >> >> Hi Ryan >> >> There are three points where 'java.lang.IllegalStateException: >> Endpoint not acquired / already released' could be triggered. Is there >> any chance I could see the complete exception stack trace (redacted if >> necessary). >> >> Oleg >> >> >> > I have a suspicion that this user is having some sort of resource >> > exhaustion issue (file descriptors? ephemeral ports?) which in turn >> > is >> > triggering some failure mode in the client that causes the connection >> > pool >> > to lock up. Thoughts? >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >>
