[
https://issues.apache.org/jira/browse/SOLR-18564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18564:
----------------------------------
Labels: pull-request-available (was: )
> CoreContainer.shutdown: a client close pool timeout skips the remaining
> closes and leaks metrics contexts
> ---------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18564
> URL: https://issues.apache.org/jira/browse/SOLR-18564
> Project: Solr
> Issue Type: Test
> Reporter: Nick Shanin
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> 🤖 _AI text below_ 🤖 _(posted on behalf of Nick Shanin)_
> *Summary: when the client close pool in CoreContainer.shutdown() times out,
> its throw skips the handler close and the root metrics context close that
> follow it, so the container's metrics contexts leak and are charged to an
> unrelated test suite.*
> This surfaced as two class teardown failures in one Crave CI run ([PR #5030
> run|https://github.com/apache/solr/actions/runs/39699700526/job/119565807999],
> seed {{A35B8669BA78E261}}). Every test passed. {{CloudHttp2SolrClientTest}}
> failed in {{tearDownAfterClass}} with "Error shutting down
> MiniSolrCloudCluster", and {{TestJavaBinCodec}} was charged by the
> ObjectReleaseTracker with 16 unreleased {{SolrMetricsContext}} objects that
> the first suite's cluster had created. The two failures are one event.
> The mechanism, from the run log and the code: {{CoreContainer.shutdown()}}
> waits on the close pool through {{ExecutorUtil.awaitTermination}}
> ({{CoreContainer.java:1314}}). When the waits expire, the utility force-stops
> the pool and throws {{RuntimeException: Timeout waiting for pool to
> shutdown}}. The throw aborts {{shutdown()}} before
> {{containerHandlers.close()}} (line 1362) and the root metrics context close
> (line 1366), which are the only release calls for those contexts. The leaked
> contexts stay in the JVM-global tracker and land on the next suite in the
> fork whose teardown checks it, which is why the charge appears on a suite
> that created none of them.
> The failure is intermittent in CI (the full solrj suite passed twice locally
> under the same seed), but it reproduces deterministically by blocking a close
> on a latch and letting the real timeout budget run out: on current code that
> fails with the same throw and an ObjectTracker leak report. A fix is prepared
> on [this
> branch|https://github.com/nick-boss-tech/solr/tree/corecontainer-shutdown-closes]:
> catch and log the timeout throw at the call site so the trailing closes
> always run, with a regression test that fails without the fix and passes with
> it. {{ExecutorUtil}} is unchanged; its throw is a contract other callers rely
> on.
> Out of scope here: why the close exceeded its budget under CI load in the
> first place. This ticket covers the skipped closes, not the slowness.
> _AI-generated description. Nick Shanin directed the work and takes
> responsibility for this contribution._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
