[JENKINS] Solr » Solr-NightlyTests-main - Build # 1160 - Still Unstable!

[Apache Jenkins Server]

Build: https://ci-builds.apache.org/job/Solr/job/Solr-NightlyTests-main/1160/

7 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
java.lang.AssertionError: expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
        at 
__randomizedtesting.SeedInfo.seed([9663F785AE6439C:813200A2F41A2E64]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at 
org.apache.solr.cloud.AbstractChaosMonkeySafeLeaderTestBase.test(AbstractChaosMonkeySafeLeaderTestBase.java:166)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1069)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1040)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:829)


FAILED:  org.apache.solr.cloud.RecoveryStrategyStressTest.stressTestRecovery

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([9663F785AE6439C]:0)


FAILED:  org.apache.solr.cloud.RecoveryStrategyStressTest.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([9663F785AE6439C]:0)


FAILED:  org.apache.solr.cloud.RecoveryStrategyStressTest.classMethod

Error Message:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=12356, name=Thread-1596, state=RUNNABLE, 
group=TGRP-RecoveryStrategyStressTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=12356, name=Thread-1596, state=RUNNABLE, 
group=TGRP-RecoveryStrategyStressTest]
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Timeout 
waiting for pool 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@4c5c7cf4[Shutting
 down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 
3] to shutdown.
        at __randomizedtesting.SeedInfo.seed([9663F785AE6439C]:0)
        at 
org.apache.solr.cloud.RecoveryStrategyStressTest.lambda$stressTestRecovery$1(RecoveryStrategyStressTest.java:80)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Timeout waiting for pool 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@4c5c7cf4[Shutting
 down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 
3] to shutdown.
        at 
org.apache.solr.common.util.ExecutorUtil.awaitTermination(ExecutorUtil.java:165)
        at 
org.apache.solr.common.util.ExecutorUtil.awaitTermination(ExecutorUtil.java:152)
        at 
org.apache.solr.common.util.ExecutorUtil.shutdownAndAwaitTermination(ExecutorUtil.java:118)
        at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:1411)
        at 
org.apache.solr.servlet.CoreContainerProvider.close(CoreContainerProvider.java:180)
        at 
org.apache.solr.servlet.CoreContainerProvider.contextDestroyed(CoreContainerProvider.java:114)
        at 
org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:1059)
        at 
org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:636)
        at 
org.eclipse.jetty.server.handler.ContextHandler.contextDestroyed(ContextHandler.java:1016)
        at 
org.eclipse.jetty.servlet.ServletHandler.doStop(ServletHandler.java:306)
        at 
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
        at 
org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
        at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
        at 
org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
        at 
org.eclipse.jetty.server.session.SessionHandler.doStop(SessionHandler.java:497)
        at 
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
        at 
org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
        at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
        at 
org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
        at 
org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:1039)
        at 
org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:399)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:1087)
        at 
org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:312)
        at 
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
        at 
org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
        at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
        at 
org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
        at 
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
        at 
org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
        at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
        at 
org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStop(GzipHandler.java:227)
        at 
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
        at 
org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
        at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
        at 
org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
        at org.eclipse.jetty.server.Server.doStop(Server.java:517)
        at 
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
        at 
org.apache.solr.embedded.JettySolrRunner.stop(JettySolrRunner.java:632)
        at 
org.apache.solr.cloud.RecoveryStrategyStressTest.lambda$stressTestRecovery$1(RecoveryStrategyStressTest.java:74)
        ... 1 more


FAILED:  
org.apache.solr.cloud.SplitShardWithNodeRoleTest.testSolrClusterWithNodeRoleWithPull

Error Message:
java.lang.AssertionError: Timed out waiting for sub shards to be active. Number 
of active shards=1
Timeout waiting to see state for collection=coll_NRT_PULL 
:DocCollection(coll_NRT_PULL//collections/coll_NRT_PULL/state.json/13 
{/collections/coll_NRT_PULL/state.json/[28]: [core_node10:2:A, core_node4:1:A, 
core_node2:2:A:L, core_node8:2:A:L, core_node9:2:A, core_node7:2:A:L]})={
  "pullReplicas":1,
  "configName":"conf",
  "replicationFactor":1,
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0,
  "perReplicaState":true,
  "shards":{
    "shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"coll_NRT_PULL_shard1_replica_n1",
          "node_name":"127.0.0.1:41503_solr",
          "type":"NRT",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41503/solr"},
        "core_node4":{
          "core":"coll_NRT_PULL_shard1_replica_p3",
          "node_name":"127.0.0.1:33471_solr",
          "type":"PULL",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:33471/solr"}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1717854736342927077",
      "shard_parent_node":"127.0.0.1:41503_solr",
      "replicas":{
        "core_node7":{
          "core":"coll_NRT_PULL_shard1_0_replica_n5",
          "node_name":"127.0.0.1:41503_solr",
          "type":"NRT",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41503/solr"},
        "core_node9":{
          "core":"coll_NRT_PULL_shard1_0_replica_p0",
          "node_name":"127.0.0.1:33471_solr",
          "type":"PULL",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:33471/solr"}},
      "range":"80000000-ffffffff",
      "state":"recovery",
      "shard_parent_zk_session":"72368324152393736"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1717854736342903249",
      "shard_parent_node":"127.0.0.1:41503_solr",
      "replicas":{
        "core_node8":{
          "core":"coll_NRT_PULL_shard1_1_replica_n6",
          "node_name":"127.0.0.1:41503_solr",
          "type":"NRT",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41503/solr"},
        "core_node10":{
          "core":"coll_NRT_PULL_shard1_1_replica_p1",
          "node_name":"127.0.0.1:33471_solr",
          "type":"PULL",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:33471/solr"}},
      "range":"0-7fffffff",
      "state":"recovery",
      "shard_parent_zk_session":"72368324152393736"}}}
Live Nodes: [127.0.0.1:33471_solr, 127.0.0.1:35603_solr, 127.0.0.1:36341_solr, 
127.0.0.1:41503_solr, 127.0.0.1:46297_solr, 127.0.0.1:46711_solr]
Last available state: 
DocCollection(coll_NRT_PULL//collections/coll_NRT_PULL/state.json/13 
{/collections/coll_NRT_PULL/state.json/[28]: [core_node10:2:A, core_node4:1:A, 
core_node2:2:A:L, core_node8:2:A:L, core_node9:2:A, core_node7:2:A:L]})={
  "pullReplicas":1,
  "configName":"conf",
  "replicationFactor":1,
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0,
  "perReplicaState":true,
  "shards":{
    "shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"coll_NRT_PULL_shard1_replica_n1",
          "node_name":"127.0.0.1:41503_solr",
          "type":"NRT",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41503/solr"},
        "core_node4":{
          "core":"coll_NRT_PULL_shard1_replica_p3",
          "node_name":"127.0.0.1:33471_solr",
          "type":"PULL",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:33471/solr"}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1717854736342927077",
      "shard_parent_node":"127.0.0.1:41503_solr",
      "replicas":{
        "core_node7":{
          "core":"coll_NRT_PULL_shard1_0_replica_n5",
          "node_name":"127.0.0.1:41503_solr",
          "type":"NRT",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41503/solr"},
        "core_node9":{
          "core":"coll_NRT_PULL_shard1_0_replica_p0",
          "node_name":"127.0.0.1:33471_solr",
          "type":"PULL",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:33471/solr"}},
      "range":"80000000-ffffffff",
      "state":"recovery",
      "shard_parent_zk_session":"72368324152393736"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1717854736342903249",
      "shard_parent_node":"127.0.0.1:41503_solr",
      "replicas":{
        "core_node8":{
          "core":"coll_NRT_PULL_shard1_1_replica_n6",
          "node_name":"127.0.0.1:41503_solr",
          "type":"NRT",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41503/solr"},
        "core_node10":{
          "core":"coll_NRT_PULL_shard1_1_replica_p1",
          "node_name":"127.0.0.1:33471_solr",
          "type":"PULL",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:33471/solr"}},
      "range":"0-7fffffff",
      "state":"recovery",
      "shard_parent_zk_session":"72368324152393736"}}}

Stack Trace:
java.lang.AssertionError: Timed out waiting for sub shards to be active. Number 
of active shards=1
Timeout waiting to see state for collection=coll_NRT_PULL 
:DocCollection(coll_NRT_PULL//collections/coll_NRT_PULL/state.json/13 
{/collections/coll_NRT_PULL/state.json/[28]: [core_node10:2:A, core_node4:1:A, 
core_node2:2:A:L, core_node8:2:A:L, core_node9:2:A, core_node7:2:A:L]})={
  "pullReplicas":1,
  "configName":"conf",
  "replicationFactor":1,
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0,
  "perReplicaState":true,
  "shards":{
    "shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"coll_NRT_PULL_shard1_replica_n1",
          "node_name":"127.0.0.1:41503_solr",
          "type":"NRT",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41503/solr"},
        "core_node4":{
          "core":"coll_NRT_PULL_shard1_replica_p3",
          "node_name":"127.0.0.1:33471_solr",
          "type":"PULL",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:33471/solr"}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1717854736342927077",
      "shard_parent_node":"127.0.0.1:41503_solr",
      "replicas":{
        "core_node7":{
          "core":"coll_NRT_PULL_shard1_0_replica_n5",
          "node_name":"127.0.0.1:41503_solr",
          "type":"NRT",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41503/solr"},
        "core_node9":{
          "core":"coll_NRT_PULL_shard1_0_replica_p0",
          "node_name":"127.0.0.1:33471_solr",
          "type":"PULL",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:33471/solr"}},
      "range":"80000000-ffffffff",
      "state":"recovery",
      "shard_parent_zk_session":"72368324152393736"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1717854736342903249",
      "shard_parent_node":"127.0.0.1:41503_solr",
      "replicas":{
        "core_node8":{
          "core":"coll_NRT_PULL_shard1_1_replica_n6",
          "node_name":"127.0.0.1:41503_solr",
          "type":"NRT",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41503/solr"},
        "core_node10":{
          "core":"coll_NRT_PULL_shard1_1_replica_p1",
          "node_name":"127.0.0.1:33471_solr",
          "type":"PULL",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:33471/solr"}},
      "range":"0-7fffffff",
      "state":"recovery",
      "shard_parent_zk_session":"72368324152393736"}}}
Live Nodes: [127.0.0.1:33471_solr, 127.0.0.1:35603_solr, 127.0.0.1:36341_solr, 
127.0.0.1:41503_solr, 127.0.0.1:46297_solr, 127.0.0.1:46711_solr]
Last available state: 
DocCollection(coll_NRT_PULL//collections/coll_NRT_PULL/state.json/13 
{/collections/coll_NRT_PULL/state.json/[28]: [core_node10:2:A, core_node4:1:A, 
core_node2:2:A:L, core_node8:2:A:L, core_node9:2:A, core_node7:2:A:L]})={
  "pullReplicas":1,
  "configName":"conf",
  "replicationFactor":1,
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0,
  "perReplicaState":true,
  "shards":{
    "shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"coll_NRT_PULL_shard1_replica_n1",
          "node_name":"127.0.0.1:41503_solr",
          "type":"NRT",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41503/solr"},
        "core_node4":{
          "core":"coll_NRT_PULL_shard1_replica_p3",
          "node_name":"127.0.0.1:33471_solr",
          "type":"PULL",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:33471/solr"}}},
    "shard1_0":{
      "parent":"shard1",
      "stateTimestamp":"1717854736342927077",
      "shard_parent_node":"127.0.0.1:41503_solr",
      "replicas":{
        "core_node7":{
          "core":"coll_NRT_PULL_shard1_0_replica_n5",
          "node_name":"127.0.0.1:41503_solr",
          "type":"NRT",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41503/solr"},
        "core_node9":{
          "core":"coll_NRT_PULL_shard1_0_replica_p0",
          "node_name":"127.0.0.1:33471_solr",
          "type":"PULL",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:33471/solr"}},
      "range":"80000000-ffffffff",
      "state":"recovery",
      "shard_parent_zk_session":"72368324152393736"},
    "shard1_1":{
      "parent":"shard1",
      "stateTimestamp":"1717854736342903249",
      "shard_parent_node":"127.0.0.1:41503_solr",
      "replicas":{
        "core_node8":{
          "core":"coll_NRT_PULL_shard1_1_replica_n6",
          "node_name":"127.0.0.1:41503_solr",
          "type":"NRT",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:41503/solr"},
        "core_node10":{
          "core":"coll_NRT_PULL_shard1_1_replica_p1",
          "node_name":"127.0.0.1:33471_solr",
          "type":"PULL",
          "force_set_state":"false",
          "base_url":"https://127.0.0.1:33471/solr"}},
      "range":"0-7fffffff",
      "state":"recovery",
      "shard_parent_zk_session":"72368324152393736"}}}
        at 
__randomizedtesting.SeedInfo.seed([9663F785AE6439C:D3C0AD5CFE280EE1]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:199)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:165)
        at 
org.apache.solr.cloud.SplitShardWithNodeRoleTest.doSplit(SplitShardWithNodeRoleTest.java:94)
        at 
org.apache.solr.cloud.SplitShardWithNodeRoleTest.testSolrClusterWithNodeRoleWithPull(SplitShardWithNodeRoleTest.java:68)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:829)


FAILED:  org.apache.solr.hdfs.cloud.HdfsRestartWhileUpdatingTest.classMethod

Error Message:
java.lang.AssertionError: ObjectTracker found 5 object(s) that were not 
released!!! [NRTCachingDirectory, NRTCachingDirectory, SolrCore, 
Http2SolrClient, NRTCachingDirectory]
org.apache.lucene.store.NRTCachingDirectory:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.lucene.store.NRTCachingDirectory
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:411)
        at 
org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:615)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1130)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

org.apache.lucene.store.NRTCachingDirectory:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.lucene.store.NRTCachingDirectory
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:411)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:420)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:865)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1134)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

org.apache.solr.core.SolrCore:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.solr.core.SolrCore
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1234)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

org.apache.solr.client.solrj.impl.Http2SolrClient:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.solr.client.solrj.impl.Http2SolrClient
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:143)
        at 
org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:978)
        at 
org.apache.solr.handler.IndexFetcher.createSolrClient(IndexFetcher.java:275)
        at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:321)
        at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:463)
        at 
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:243)
        at 
org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:701)
        at 
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:212)
        at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:363)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.Exception: Submitter stack trace
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:330)
        at 
java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
        at 
com.codahale.metrics.InstrumentedExecutorService.submit(InstrumentedExecutorService.java:104)
        at 
org.apache.solr.update.DefaultSolrCoreState$2.run(DefaultSolrCoreState.java:349)
        ... 7 more

org.apache.lucene.store.NRTCachingDirectory:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.lucene.store.NRTCachingDirectory
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:411)
        at 
org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:108)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:904)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1134)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

 expected null, but was:<ObjectTracker found 5 object(s) that were not 
released!!! [NRTCachingDirectory, NRTCachingDirectory, SolrCore, 
Http2SolrClient, NRTCachingDirectory]
org.apache.lucene.store.NRTCachingDirectory:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.lucene.store.NRTCachingDirectory
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:411)
        at 
org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:615)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1130)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

org.apache.lucene.store.NRTCachingDirectory:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.lucene.store.NRTCachingDirectory
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:411)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:420)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:865)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1134)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

org.apache.solr.core.SolrCore:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.solr.core.SolrCore
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1234)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

org.apache.solr.client.solrj.impl.Http2SolrClient:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.solr.client.solrj.impl.Http2SolrClient
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:143)
        at 
org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:978)
        at 
org.apache.solr.handler.IndexFetcher.createSolrClient(IndexFetcher.java:275)
        at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:321)
        at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:463)
        at 
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:243)
        at 
org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:701)
        at 
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:212)
        at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:363)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.Exception: Submitter stack trace
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:330)
        at 
java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
        at 
com.codahale.metrics.InstrumentedExecutorService.submit(InstrumentedExecutorService.java:104)
        at 
org.apache.solr.update.DefaultSolrCoreState$2.run(DefaultSolrCoreState.java:349)
        ... 7 more

org.apache.lucene.store.NRTCachingDirectory:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.lucene.store.NRTCachingDirectory
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:411)
        at 
org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:108)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:904)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1134)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

>

Stack Trace:
java.lang.AssertionError: ObjectTracker found 5 object(s) that were not 
released!!! [NRTCachingDirectory, NRTCachingDirectory, SolrCore, 
Http2SolrClient, NRTCachingDirectory]
org.apache.lucene.store.NRTCachingDirectory:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.lucene.store.NRTCachingDirectory
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:411)
        at 
org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:615)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1130)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

org.apache.lucene.store.NRTCachingDirectory:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.lucene.store.NRTCachingDirectory
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:411)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:420)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:865)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1134)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

org.apache.solr.core.SolrCore:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.solr.core.SolrCore
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1234)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

org.apache.solr.client.solrj.impl.Http2SolrClient:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.solr.client.solrj.impl.Http2SolrClient
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:143)
        at 
org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:978)
        at 
org.apache.solr.handler.IndexFetcher.createSolrClient(IndexFetcher.java:275)
        at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:321)
        at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:463)
        at 
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:243)
        at 
org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:701)
        at 
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:212)
        at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:363)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.Exception: Submitter stack trace
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:330)
        at 
java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
        at 
com.codahale.metrics.InstrumentedExecutorService.submit(InstrumentedExecutorService.java:104)
        at 
org.apache.solr.update.DefaultSolrCoreState$2.run(DefaultSolrCoreState.java:349)
        ... 7 more

org.apache.lucene.store.NRTCachingDirectory:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.lucene.store.NRTCachingDirectory
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:411)
        at 
org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:108)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:904)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1134)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

 expected null, but was:<ObjectTracker found 5 object(s) that were not 
released!!! [NRTCachingDirectory, NRTCachingDirectory, SolrCore, 
Http2SolrClient, NRTCachingDirectory]
org.apache.lucene.store.NRTCachingDirectory:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.lucene.store.NRTCachingDirectory
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
        at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:411)
        at 
org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:615)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1130)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1056)
        at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1743)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1609)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:118)
        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:408)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:374)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:231)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:238)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:955)
        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:903)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)
        at 
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:243)
        at 
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:204)
        at 
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:247)
        at 
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:211)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:198)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:182)
        at 
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at 
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at 
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:829)

org.apache.lucene.store.NRTCachingDirectory:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException:
 org.apache.lucene.store.NRTCachingDirectory
        at org

[...truncated too long message...]

java:36)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at 
app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base@11.0.16.1/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.TimeoutException
        at 
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:214)
        at 
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:502)
        ... 52 more




Build Log:
[...truncated 102855 lines...]
ERROR: The following test(s) have failed:
  - org.apache.solr.client.solrj.TestLBHttp2SolrClient.testTwoServers 
(:solr:solrj)
    Test history: 
https://ge.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.client.solrj.TestLBHttp2SolrClient&tests.test=testTwoServers
 
http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.client.solrj.TestLBHttp2SolrClient.testTwoServers
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.TestLBHttp2SolrClient.txt
    Reproduce with: ./gradlew :solr:solrj:test --tests 
"org.apache.solr.client.solrj.TestLBHttp2SolrClient.testTwoServers" 
-Ptests.jvms=4 -Ptests.haltonfailure=false 
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC 
-XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=9663F785AE6439C -Ptests.multiplier=2 -Ptests.nightly=true 
-Ptests.badapples=false -Ptests.file.encoding=UTF-8 
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.hdfs.cloud.HdfsRestartWhileUpdatingTest (:solr:modules:hdfs)
    Test history: 
https://ge.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.HdfsRestartWhileUpdatingTest
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.HdfsRestartWhileUpdatingTest.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests 
"org.apache.solr.hdfs.cloud.HdfsRestartWhileUpdatingTest" -Ptests.jvms=4 
-Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 
-XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=9663F785AE6439C -Ptests.multiplier=2 -Ptests.nightly=true 
-Ptests.badapples=false -Ptests.file.encoding=UTF-8 
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test (:solr:core)
    Test history: 
https://ge.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.cloud.ChaosMonkeySafeLeaderTest&tests.test=test
 
http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.txt
    Reproduce with: ./gradlew :solr:core:test --tests 
"org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test" -Ptests.jvms=4 
-Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 
-XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=9663F785AE6439C -Ptests.multiplier=2 -Ptests.nightly=true 
-Ptests.badapples=false -Ptests.file.encoding=UTF-8 
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - 
org.apache.solr.cloud.SplitShardWithNodeRoleTest.testSolrClusterWithNodeRoleWithPull
 (:solr:core)
    Test history: 
https://ge.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.cloud.SplitShardWithNodeRoleTest&tests.test=testSolrClusterWithNodeRoleWithPull
 
http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.SplitShardWithNodeRoleTest.testSolrClusterWithNodeRoleWithPull
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.SplitShardWithNodeRoleTest.txt
    Reproduce with: ./gradlew :solr:core:test --tests 
"org.apache.solr.cloud.SplitShardWithNodeRoleTest.testSolrClusterWithNodeRoleWithPull"
 -Ptests.jvms=4 -Ptests.haltonfailure=false 
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC 
-XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=9663F785AE6439C -Ptests.multiplier=2 -Ptests.nightly=true 
-Ptests.badapples=false -Ptests.file.encoding=UTF-8 
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.cloud.RecoveryStrategyStressTest.stressTestRecovery 
(:solr:core)
    Test history: 
https://ge.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.cloud.RecoveryStrategyStressTest&tests.test=stressTestRecovery
 
http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.RecoveryStrategyStressTest.stressTestRecovery
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.RecoveryStrategyStressTest.txt
    Reproduce with: ./gradlew :solr:core:test --tests 
"org.apache.solr.cloud.RecoveryStrategyStressTest.stressTestRecovery" 
-Ptests.jvms=4 -Ptests.haltonfailure=false 
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC 
-XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=9663F785AE6439C -Ptests.multiplier=2 -Ptests.nightly=true 
-Ptests.badapples=false -Ptests.file.encoding=UTF-8 
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.cloud.RecoveryStrategyStressTest (:solr:core)
    Test history: 
https://ge.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.cloud.RecoveryStrategyStressTest
    Test output: 
/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.RecoveryStrategyStressTest.txt
    Reproduce with: ./gradlew :solr:core:test --tests 
"org.apache.solr.cloud.RecoveryStrategyStressTest" -Ptests.jvms=4 
-Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 
-XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" 
-Ptests.seed=9663F785AE6439C -Ptests.multiplier=2 -Ptests.nightly=true 
-Ptests.badapples=false -Ptests.file.encoding=UTF-8 
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt


Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.4/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 2h 41m 45s
141 actionable tasks: 141 executed

Publishing build scan...
https://ge.apache.org/s/3e73drzs6a5ju

Build step 'Invoke Gradle script' changed build result to SUCCESS
Archiving artifacts
‘**/*.events’ doesn’t match anything: ‘**’ exists but not ‘**/*.events’
No artifacts found that match the file pattern 
"**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
[Checks API] No suitable checks publisher found.
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscr...@solr.apache.org
For additional commands, e-mail: builds-h...@solr.apache.org