We've reverted to Apache 4 for now since we're continuing to see reports of this issue trickle in. Here's some more information I received:
> I can tell you though what was triggering it. We had priming that sent several requests to the service at the exact same second, so we had up to 500 requests to that dependency at same time (our hosts are pretty big - 48 physical cores) which I believe led to some race condition when freeing resources. Once we changed our priming so it’s evenly distributed over time we stopped seeing the issue. On Wed, Mar 24, 2021 at 11:44 AM Ryan Schmitt <[email protected]> wrote: > OK, I'll try to get more diagnostics next week when I get back from > vacation. > > On Wed, Mar 24, 2021 at 11:42 AM Oleg Kalnichevski <[email protected]> > wrote: > >> On Wed, 2021-03-24 at 11:38 -0700, Ryan Schmitt wrote: >> > But how does that explain the client/connection pool locking up and >> > failing all requests until the service is restarted? >> > >> >> It does not. Just an exception stack trace is not enough to make any >> semi-educated guesses here. Thread dump would likely help more. >> >> Oleg >> >> >> > On Wed, Mar 24, 2021 at 10:34 AM Oleg Kalnichevski <[email protected]> >> > wrote: >> > > >> > > On 3/23/2021 11:13 PM, Ryan Schmitt wrote: >> > > > Oh, I actually *do* have more: >> > > > >> > > > Internal Failure java.lang.IllegalStateException: Endpoint not >> > > acquired / >> > > > already released >> > > > at >> > > > >> > > com.amazon.coral.apache.hc.client5.http.impl.classic.InternalExecRu >> > > ntime.ensureValid(InternalExecRuntime.java:142) >> > > > >> > > > at >> > > > >> > > com.amazon.coral.apache.hc.client5.http.impl.classic.InternalExecRu >> > > ntime.connectEndpoint(InternalExecRuntime.java:172) >> > > > >> > > >> > > Hi Ryan >> > > >> > > It looks like this is most likely to be happening when a request >> > > being >> > > executed gets aborted from another execution thread. >> > > >> > > Oleg >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >>
