Nick Shanin created SOLR-18564:
----------------------------------
Summary: CoreContainer.shutdown: a client close pool timeout skips
the remaining closes and leaks metrics contexts
Key: SOLR-18564
URL: https://issues.apache.org/jira/browse/SOLR-18564
Project: Solr
Issue Type: Test
Reporter: Nick Shanin
🤖 _AI text below_ 🤖 _(posted on behalf of Nick Shanin)_
*Summary: when the client close pool in CoreContainer.shutdown() times out, its
throw skips the handler close and the root metrics context close that follow
it, so the container's metrics contexts leak and are charged to an unrelated
test suite.*
This surfaced as two class teardown failures in one Crave CI run ([PR #5030
run|https://github.com/apache/solr/actions/runs/39699700526/job/119565807999],
seed {{A35B8669BA78E261}}). Every test passed. {{CloudHttp2SolrClientTest}}
failed in {{tearDownAfterClass}} with "Error shutting down
MiniSolrCloudCluster", and {{TestJavaBinCodec}} was charged by the
ObjectReleaseTracker with 16 unreleased {{SolrMetricsContext}} objects that the
first suite's cluster had created. The two failures are one event.
The mechanism, from the run log and the code: {{CoreContainer.shutdown()}}
waits on the close pool through {{ExecutorUtil.awaitTermination}}
({{CoreContainer.java:1314}}). When the waits expire, the utility force-stops
the pool and throws {{RuntimeException: Timeout waiting for pool to shutdown}}.
The throw aborts {{shutdown()}} before {{containerHandlers.close()}} (line
1362) and the root metrics context close (line 1366), which are the only
release calls for those contexts. The leaked contexts stay in the JVM-global
tracker and land on the next suite in the fork whose teardown checks it, which
is why the charge appears on a suite that created none of them.
The failure is intermittent in CI (the full solrj suite passed twice locally
under the same seed), but it reproduces deterministically by blocking a close
on a latch and letting the real timeout budget run out: on current code that
fails with the same throw and an ObjectTracker leak report. A fix is prepared
on [this
branch|https://github.com/nick-boss-tech/solr/tree/corecontainer-shutdown-closes]:
catch and log the timeout throw at the call site so the trailing closes always
run, with a regression test that fails without the fix and passes with it.
{{ExecutorUtil}} is unchanged; its throw is a contract other callers rely on.
Out of scope here: why the close exceeded its budget under CI load in the first
place. This ticket covers the skipped closes, not the slowness.
_AI-generated description. Nick Shanin directed the work and takes
responsibility for this contribution._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
