Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/6403/
Java: 64bit/jdk-11.0.6 -XX:+UseCompressedOops -XX:+UseParallelGC

16 tests failed.
FAILED:  org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test

Error Message:
Node 127.0.0.1:56003_solr has 4 replicas. Expected num replicas : 2. state:  
DocCollection(hdfsbackuprestore_testok_restored/17)={   "pullReplicas":1,   
"replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",  
     "state":"active",       "replicas":{         "core_node2":{           
"core":"hdfsbackuprestore_testok_restored_shard2_replica_n1",           
"leader":"true",           "core_node_name":"core_node2",           
"base_url":"https://127.0.0.1:56003/solr";,           
"node_name":"127.0.0.1:56003_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard2",   
        "type":"NRT",           "force_set_state":"false"},         
"core_node6":{           
"core":"hdfsbackuprestore_testok_restored_shard2_replica_p5",           
"core_node_name":"core_node6",           
"base_url":"https://127.0.0.1:56003/solr";,           
"node_name":"127.0.0.1:56003_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard2",   
        "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156257117561466"},     "shard1":{       
"range":"80000000-ffffffff",       "state":"active",       "replicas":{         
"core_node4":{           
"core":"hdfsbackuprestore_testok_restored_shard1_replica_n3",           
"leader":"true",           "core_node_name":"core_node4",           
"base_url":"https://127.0.0.1:56003/solr";,           
"node_name":"127.0.0.1:56003_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard1",   
        "type":"NRT",           "force_set_state":"false"},         
"core_node8":{           
"core":"hdfsbackuprestore_testok_restored_shard1_replica_p7",           
"core_node_name":"core_node8",           
"base_url":"https://127.0.0.1:56003/solr";,           
"node_name":"127.0.0.1:56003_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard1",   
        "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156257117657404"}},   "router":{"name":"compositeId"},   
"nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:56003_solr has 4 replicas. Expected 
num replicas : 2. state: 
DocCollection(hdfsbackuprestore_testok_restored/17)={
  "pullReplicas":1,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"hdfsbackuprestore_testok_restored_shard2_replica_n1",
          "leader":"true",
          "core_node_name":"core_node2",
          "base_url":"https://127.0.0.1:56003/solr";,
          "node_name":"127.0.0.1:56003_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard2",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node6":{
          "core":"hdfsbackuprestore_testok_restored_shard2_replica_p5",
          "core_node_name":"core_node6",
          "base_url":"https://127.0.0.1:56003/solr";,
          "node_name":"127.0.0.1:56003_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard2",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156257117561466"},
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"hdfsbackuprestore_testok_restored_shard1_replica_n3",
          "leader":"true",
          "core_node_name":"core_node4",
          "base_url":"https://127.0.0.1:56003/solr";,
          "node_name":"127.0.0.1:56003_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard1",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node8":{
          "core":"hdfsbackuprestore_testok_restored_shard1_replica_p7",
          "core_node_name":"core_node8",
          "base_url":"https://127.0.0.1:56003/solr";,
          "node_name":"127.0.0.1:56003_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard1",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156257117657404"}},
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0}
        at 
__randomizedtesting.SeedInfo.seed([B4A5292B78D40EF3:3CF116F1D628630B]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:374)
        at java.base/java.util.HashMap.forEach(HashMap.java:1336)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:373)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:155)
        at 
org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test(TestHdfsCloudBackupRestore.java:225)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
        at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test

Error Message:
Node 127.0.0.1:56007_solr has 4 replicas. Expected num replicas : 2. state:  
DocCollection(hdfsbackuprestore_testok_restored/17)={   "pullReplicas":1,   
"replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",  
     "state":"active",       "replicas":{         "core_node2":{           
"core":"hdfsbackuprestore_testok_restored_shard2_replica_n1",           
"leader":"true",           "core_node_name":"core_node2",           
"base_url":"https://127.0.0.1:56007/solr";,           
"node_name":"127.0.0.1:56007_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard2",   
        "type":"NRT",           "force_set_state":"false"},         
"core_node6":{           
"core":"hdfsbackuprestore_testok_restored_shard2_replica_p5",           
"core_node_name":"core_node6",           
"base_url":"https://127.0.0.1:56007/solr";,           
"node_name":"127.0.0.1:56007_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard2",   
        "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156258677280263"},     "shard1":{       
"range":"80000000-ffffffff",       "state":"active",       "replicas":{         
"core_node4":{           
"core":"hdfsbackuprestore_testok_restored_shard1_replica_n3",           
"leader":"true",           "core_node_name":"core_node4",           
"base_url":"https://127.0.0.1:56007/solr";,           
"node_name":"127.0.0.1:56007_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard1",   
        "type":"NRT",           "force_set_state":"false"},         
"core_node8":{           
"core":"hdfsbackuprestore_testok_restored_shard1_replica_p7",           
"core_node_name":"core_node8",           
"base_url":"https://127.0.0.1:56007/solr";,           
"node_name":"127.0.0.1:56007_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard1",   
        "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156258677319232"}},   "router":{"name":"compositeId"},   
"nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:56007_solr has 4 replicas. Expected 
num replicas : 2. state: 
DocCollection(hdfsbackuprestore_testok_restored/17)={
  "pullReplicas":1,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"hdfsbackuprestore_testok_restored_shard2_replica_n1",
          "leader":"true",
          "core_node_name":"core_node2",
          "base_url":"https://127.0.0.1:56007/solr";,
          "node_name":"127.0.0.1:56007_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard2",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node6":{
          "core":"hdfsbackuprestore_testok_restored_shard2_replica_p5",
          "core_node_name":"core_node6",
          "base_url":"https://127.0.0.1:56007/solr";,
          "node_name":"127.0.0.1:56007_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard2",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156258677280263"},
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"hdfsbackuprestore_testok_restored_shard1_replica_n3",
          "leader":"true",
          "core_node_name":"core_node4",
          "base_url":"https://127.0.0.1:56007/solr";,
          "node_name":"127.0.0.1:56007_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard1",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node8":{
          "core":"hdfsbackuprestore_testok_restored_shard1_replica_p7",
          "core_node_name":"core_node8",
          "base_url":"https://127.0.0.1:56007/solr";,
          "node_name":"127.0.0.1:56007_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard1",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156258677319232"}},
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0}
        at 
__randomizedtesting.SeedInfo.seed([B4A5292B78D40EF3:3CF116F1D628630B]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:374)
        at java.base/java.util.HashMap.forEach(HashMap.java:1336)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:373)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:155)
        at 
org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test(TestHdfsCloudBackupRestore.java:225)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
        at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test

Error Message:
Node 127.0.0.1:56010_solr has 4 replicas. Expected num replicas : 2. state:  
DocCollection(hdfsbackuprestore_testok_restored/16)={   "pullReplicas":1,   
"replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",  
     "state":"active",       "replicas":{         "core_node2":{           
"core":"hdfsbackuprestore_testok_restored_shard2_replica_n1",           
"leader":"true",           "core_node_name":"core_node2",           
"base_url":"https://127.0.0.1:56010/solr";,           
"node_name":"127.0.0.1:56010_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard2",   
        "type":"NRT",           "force_set_state":"false"},         
"core_node6":{           
"core":"hdfsbackuprestore_testok_restored_shard2_replica_p5",           
"core_node_name":"core_node6",           
"base_url":"https://127.0.0.1:56010/solr";,           
"node_name":"127.0.0.1:56010_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard2",   
        "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156255289405919"},     "shard1":{       
"range":"80000000-ffffffff",       "state":"active",       "replicas":{         
"core_node4":{           
"core":"hdfsbackuprestore_testok_restored_shard1_replica_n3",           
"leader":"true",           "core_node_name":"core_node4",           
"base_url":"https://127.0.0.1:56010/solr";,           
"node_name":"127.0.0.1:56010_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard1",   
        "type":"NRT",           "force_set_state":"false"},         
"core_node8":{           
"core":"hdfsbackuprestore_testok_restored_shard1_replica_p7",           
"core_node_name":"core_node8",           
"base_url":"https://127.0.0.1:56010/solr";,           
"node_name":"127.0.0.1:56010_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard1",   
        "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156255289502960"}},   "router":{"name":"compositeId"},   
"nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:56010_solr has 4 replicas. Expected 
num replicas : 2. state: 
DocCollection(hdfsbackuprestore_testok_restored/16)={
  "pullReplicas":1,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"hdfsbackuprestore_testok_restored_shard2_replica_n1",
          "leader":"true",
          "core_node_name":"core_node2",
          "base_url":"https://127.0.0.1:56010/solr";,
          "node_name":"127.0.0.1:56010_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard2",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node6":{
          "core":"hdfsbackuprestore_testok_restored_shard2_replica_p5",
          "core_node_name":"core_node6",
          "base_url":"https://127.0.0.1:56010/solr";,
          "node_name":"127.0.0.1:56010_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard2",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156255289405919"},
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"hdfsbackuprestore_testok_restored_shard1_replica_n3",
          "leader":"true",
          "core_node_name":"core_node4",
          "base_url":"https://127.0.0.1:56010/solr";,
          "node_name":"127.0.0.1:56010_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard1",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node8":{
          "core":"hdfsbackuprestore_testok_restored_shard1_replica_p7",
          "core_node_name":"core_node8",
          "base_url":"https://127.0.0.1:56010/solr";,
          "node_name":"127.0.0.1:56010_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard1",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156255289502960"}},
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0}
        at 
__randomizedtesting.SeedInfo.seed([B4A5292B78D40EF3:3CF116F1D628630B]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:374)
        at java.base/java.util.HashMap.forEach(HashMap.java:1336)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:373)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:155)
        at 
org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test(TestHdfsCloudBackupRestore.java:225)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
        at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore

Error Message:
1 thread leaked from SUITE scope at 
org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore:     1) 
Thread[id=1053, name=indexFetcher-138-thread-1, state=TIMED_WAITING, 
group=TGRP-TestHdfsCloudBackupRestore]         at 
[email protected]/java.lang.Object.wait(Native Method)         at 
app//org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:175)
         at app//org.apache.solr.core.SolrCore.close(SolrCore.java:1690)        
 at 
app//org.apache.solr.handler.IndexFetcher.openNewSearcherAndUpdateCommitPoint(IndexFetcher.java:939)
         at 
app//org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:679)
         at 
app//org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:354)
         at 
app//org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:455)
         at 
app//org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1247)
         at 
app//org.apache.solr.handler.ReplicationHandler$$Lambda$704/0x0000000800adfc40.run(Unknown
 Source)         at 
[email protected]/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
         at 
[email protected]/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore: 
   1) Thread[id=1053, name=indexFetcher-138-thread-1, state=TIMED_WAITING, 
group=TGRP-TestHdfsCloudBackupRestore]
        at [email protected]/java.lang.Object.wait(Native Method)
        at 
app//org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:175)
        at app//org.apache.solr.core.SolrCore.close(SolrCore.java:1690)
        at 
app//org.apache.solr.handler.IndexFetcher.openNewSearcherAndUpdateCommitPoint(IndexFetcher.java:939)
        at 
app//org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:679)
        at 
app//org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:354)
        at 
app//org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:455)
        at 
app//org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1247)
        at 
app//org.apache.solr.handler.ReplicationHandler$$Lambda$704/0x0000000800adfc40.run(Unknown
 Source)
        at 
[email protected]/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at 
[email protected]/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
        at __randomizedtesting.SeedInfo.seed([B4A5292B78D40EF3]:0)


FAILED:  org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test

Error Message:
Node 127.0.0.1:56019_solr has 4 replicas. Expected num replicas : 2. state:  
DocCollection(hdfsbackuprestore_testok_restored/17)={   "pullReplicas":1,   
"replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",  
     "state":"active",       "replicas":{         "core_node2":{           
"core":"hdfsbackuprestore_testok_restored_shard2_replica_n1",           
"leader":"true",           "core_node_name":"core_node2",           
"base_url":"https://127.0.0.1:56019/solr";,           
"node_name":"127.0.0.1:56019_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard2",   
        "type":"NRT",           "force_set_state":"false"},         
"core_node6":{           
"core":"hdfsbackuprestore_testok_restored_shard2_replica_p5",           
"core_node_name":"core_node6",           
"base_url":"https://127.0.0.1:56019/solr";,           
"node_name":"127.0.0.1:56019_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard2",   
        "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156257777024580"},     "shard1":{       
"range":"80000000-ffffffff",       "state":"active",       "replicas":{         
"core_node4":{           
"core":"hdfsbackuprestore_testok_restored_shard1_replica_n3",           
"leader":"true",           "core_node_name":"core_node4",           
"base_url":"https://127.0.0.1:56019/solr";,           
"node_name":"127.0.0.1:56019_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard1",   
        "type":"NRT",           "force_set_state":"false"},         
"core_node8":{           
"core":"hdfsbackuprestore_testok_restored_shard1_replica_p7",           
"core_node_name":"core_node8",           
"base_url":"https://127.0.0.1:56019/solr";,           
"node_name":"127.0.0.1:56019_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard1",   
        "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156257777063510"}},   "router":{"name":"compositeId"},   
"nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:56019_solr has 4 replicas. Expected 
num replicas : 2. state: 
DocCollection(hdfsbackuprestore_testok_restored/17)={
  "pullReplicas":1,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"hdfsbackuprestore_testok_restored_shard2_replica_n1",
          "leader":"true",
          "core_node_name":"core_node2",
          "base_url":"https://127.0.0.1:56019/solr";,
          "node_name":"127.0.0.1:56019_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard2",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node6":{
          "core":"hdfsbackuprestore_testok_restored_shard2_replica_p5",
          "core_node_name":"core_node6",
          "base_url":"https://127.0.0.1:56019/solr";,
          "node_name":"127.0.0.1:56019_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard2",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156257777024580"},
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"hdfsbackuprestore_testok_restored_shard1_replica_n3",
          "leader":"true",
          "core_node_name":"core_node4",
          "base_url":"https://127.0.0.1:56019/solr";,
          "node_name":"127.0.0.1:56019_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard1",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node8":{
          "core":"hdfsbackuprestore_testok_restored_shard1_replica_p7",
          "core_node_name":"core_node8",
          "base_url":"https://127.0.0.1:56019/solr";,
          "node_name":"127.0.0.1:56019_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard1",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156257777063510"}},
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0}
        at 
__randomizedtesting.SeedInfo.seed([B4A5292B78D40EF3:3CF116F1D628630B]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:374)
        at java.base/java.util.HashMap.forEach(HashMap.java:1336)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:373)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:155)
        at 
org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test(TestHdfsCloudBackupRestore.java:225)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
        at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore

Error Message:
1 thread leaked from SUITE scope at 
org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore:     1) 
Thread[id=1055, name=indexFetcher-140-thread-1, state=TIMED_WAITING, 
group=TGRP-TestHdfsCloudBackupRestore]         at 
[email protected]/java.lang.Object.wait(Native Method)         at 
app//org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:175)
         at app//org.apache.solr.core.SolrCore.close(SolrCore.java:1690)        
 at 
app//org.apache.solr.handler.IndexFetcher.openNewSearcherAndUpdateCommitPoint(IndexFetcher.java:939)
         at 
app//org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:679)
         at 
app//org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:354)
         at 
app//org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:455)
         at 
app//org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1247)
         at 
app//org.apache.solr.handler.ReplicationHandler$$Lambda$708/0x0000000800ad7440.run(Unknown
 Source)         at 
[email protected]/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
         at 
[email protected]/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
         at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
         at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at [email protected]/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore: 
   1) Thread[id=1055, name=indexFetcher-140-thread-1, state=TIMED_WAITING, 
group=TGRP-TestHdfsCloudBackupRestore]
        at [email protected]/java.lang.Object.wait(Native Method)
        at 
app//org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:175)
        at app//org.apache.solr.core.SolrCore.close(SolrCore.java:1690)
        at 
app//org.apache.solr.handler.IndexFetcher.openNewSearcherAndUpdateCommitPoint(IndexFetcher.java:939)
        at 
app//org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:679)
        at 
app//org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:354)
        at 
app//org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:455)
        at 
app//org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1247)
        at 
app//org.apache.solr.handler.ReplicationHandler$$Lambda$708/0x0000000800ad7440.run(Unknown
 Source)
        at 
[email protected]/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at 
[email protected]/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at 
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at 
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at [email protected]/java.lang.Thread.run(Thread.java:834)
        at __randomizedtesting.SeedInfo.seed([B4A5292B78D40EF3]:0)


FAILED:  org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test

Error Message:
Node 127.0.0.1:56000_solr has 4 replicas. Expected num replicas : 2. state:  
DocCollection(hdfsbackuprestore_testok_restored/17)={   "pullReplicas":1,   
"replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",  
     "state":"active",       "replicas":{         "core_node2":{           
"core":"hdfsbackuprestore_testok_restored_shard2_replica_n1",           
"leader":"true",           "core_node_name":"core_node2",           
"base_url":"https://127.0.0.1:56000/solr";,           
"node_name":"127.0.0.1:56000_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard2",   
        "type":"NRT",           "force_set_state":"false"},         
"core_node6":{           
"core":"hdfsbackuprestore_testok_restored_shard2_replica_p5",           
"core_node_name":"core_node6",           
"base_url":"https://127.0.0.1:56000/solr";,           
"node_name":"127.0.0.1:56000_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard2",   
        "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156255615934219"},     "shard1":{       
"range":"80000000-ffffffff",       "state":"active",       "replicas":{         
"core_node4":{           
"core":"hdfsbackuprestore_testok_restored_shard1_replica_n3",           
"leader":"true",           "core_node_name":"core_node4",           
"base_url":"https://127.0.0.1:56000/solr";,           
"node_name":"127.0.0.1:56000_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard1",   
        "type":"NRT",           "force_set_state":"false"},         
"core_node8":{           
"core":"hdfsbackuprestore_testok_restored_shard1_replica_p7",           
"core_node_name":"core_node8",           
"base_url":"https://127.0.0.1:56000/solr";,           
"node_name":"127.0.0.1:56000_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard1",   
        "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156255616089937"}},   "router":{"name":"compositeId"},   
"nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:56000_solr has 4 replicas. Expected 
num replicas : 2. state: 
DocCollection(hdfsbackuprestore_testok_restored/17)={
  "pullReplicas":1,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"hdfsbackuprestore_testok_restored_shard2_replica_n1",
          "leader":"true",
          "core_node_name":"core_node2",
          "base_url":"https://127.0.0.1:56000/solr";,
          "node_name":"127.0.0.1:56000_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard2",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node6":{
          "core":"hdfsbackuprestore_testok_restored_shard2_replica_p5",
          "core_node_name":"core_node6",
          "base_url":"https://127.0.0.1:56000/solr";,
          "node_name":"127.0.0.1:56000_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard2",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156255615934219"},
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"hdfsbackuprestore_testok_restored_shard1_replica_n3",
          "leader":"true",
          "core_node_name":"core_node4",
          "base_url":"https://127.0.0.1:56000/solr";,
          "node_name":"127.0.0.1:56000_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard1",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node8":{
          "core":"hdfsbackuprestore_testok_restored_shard1_replica_p7",
          "core_node_name":"core_node8",
          "base_url":"https://127.0.0.1:56000/solr";,
          "node_name":"127.0.0.1:56000_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard1",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156255616089937"}},
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0}
        at 
__randomizedtesting.SeedInfo.seed([B4A5292B78D40EF3:3CF116F1D628630B]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:374)
        at java.base/java.util.HashMap.forEach(HashMap.java:1336)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:373)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:155)
        at 
org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test(TestHdfsCloudBackupRestore.java:225)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
        at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test

Error Message:
Node 127.0.0.1:57051_solr has 4 replicas. Expected num replicas : 2. state:  
DocCollection(hdfsbackuprestore_testok_restored/16)={   "pullReplicas":1,   
"replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",  
     "state":"active",       "replicas":{         "core_node2":{           
"core":"hdfsbackuprestore_testok_restored_shard2_replica_n1",           
"leader":"true",           "core_node_name":"core_node2",           
"base_url":"https://127.0.0.1:57051/solr";,           
"node_name":"127.0.0.1:57051_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard2",   
        "type":"NRT",           "force_set_state":"false"},         
"core_node6":{           
"core":"hdfsbackuprestore_testok_restored_shard2_replica_p5",           
"core_node_name":"core_node6",           
"base_url":"https://127.0.0.1:57051/solr";,           
"node_name":"127.0.0.1:57051_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard2",   
        "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597154408193756193"},     "shard1":{       
"range":"80000000-ffffffff",       "state":"active",       "replicas":{         
"core_node4":{           
"core":"hdfsbackuprestore_testok_restored_shard1_replica_n3",           
"leader":"true",           "core_node_name":"core_node4",           
"base_url":"https://127.0.0.1:57051/solr";,           
"node_name":"127.0.0.1:57051_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard1",   
        "type":"NRT",           "force_set_state":"false"},         
"core_node8":{           
"core":"hdfsbackuprestore_testok_restored_shard1_replica_p7",           
"core_node_name":"core_node8",           
"base_url":"https://127.0.0.1:57051/solr";,           
"node_name":"127.0.0.1:57051_solr",           "state":"active",           
"collection":"hdfsbackuprestore_testok_restored",           "shard":"shard1",   
        "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597154408193773684"}},   "router":{"name":"compositeId"},   
"nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:57051_solr has 4 replicas. Expected 
num replicas : 2. state: 
DocCollection(hdfsbackuprestore_testok_restored/16)={
  "pullReplicas":1,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"hdfsbackuprestore_testok_restored_shard2_replica_n1",
          "leader":"true",
          "core_node_name":"core_node2",
          "base_url":"https://127.0.0.1:57051/solr";,
          "node_name":"127.0.0.1:57051_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard2",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node6":{
          "core":"hdfsbackuprestore_testok_restored_shard2_replica_p5",
          "core_node_name":"core_node6",
          "base_url":"https://127.0.0.1:57051/solr";,
          "node_name":"127.0.0.1:57051_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard2",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597154408193756193"},
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"hdfsbackuprestore_testok_restored_shard1_replica_n3",
          "leader":"true",
          "core_node_name":"core_node4",
          "base_url":"https://127.0.0.1:57051/solr";,
          "node_name":"127.0.0.1:57051_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard1",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node8":{
          "core":"hdfsbackuprestore_testok_restored_shard1_replica_p7",
          "core_node_name":"core_node8",
          "base_url":"https://127.0.0.1:57051/solr";,
          "node_name":"127.0.0.1:57051_solr",
          "state":"active",
          "collection":"hdfsbackuprestore_testok_restored",
          "shard":"shard1",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597154408193773684"}},
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0}
        at 
__randomizedtesting.SeedInfo.seed([B4A5292B78D40EF3:3CF116F1D628630B]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:374)
        at java.base/java.util.HashMap.forEach(HashMap.java:1336)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:373)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:155)
        at 
org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore.test(TestHdfsCloudBackupRestore.java:225)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
        at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  
org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Node 127.0.0.1:57671_solr has 4 replicas. Expected num replicas : 2. state:  
DocCollection(backuprestore_testok_restored/17)={   "pullReplicas":1,   
"replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",  
     "state":"active",       "replicas":{         "core_node2":{           
"core":"backuprestore_testok_restored_shard2_replica_n1",           
"leader":"true",           "core_node_name":"core_node2",           
"base_url":"http://127.0.0.1:57671/solr";,           
"node_name":"127.0.0.1:57671_solr",           "state":"active",           
"collection":"backuprestore_testok_restored",           "shard":"shard2",       
    "type":"NRT",           "force_set_state":"false"},         "core_node6":{  
         "core":"backuprestore_testok_restored_shard2_replica_p5",           
"core_node_name":"core_node6",           
"base_url":"http://127.0.0.1:57671/solr";,           
"node_name":"127.0.0.1:57671_solr",           "state":"active",           
"collection":"backuprestore_testok_restored",           "shard":"shard2",       
    "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156444717550765"},     "shard1":{       
"range":"80000000-ffffffff",       "state":"active",       "replicas":{         
"core_node4":{           
"core":"backuprestore_testok_restored_shard1_replica_n3",           
"leader":"true",           "core_node_name":"core_node4",           
"base_url":"http://127.0.0.1:57671/solr";,           
"node_name":"127.0.0.1:57671_solr",           "state":"active",           
"collection":"backuprestore_testok_restored",           "shard":"shard1",       
    "type":"NRT",           "force_set_state":"false"},         "core_node8":{  
         "core":"backuprestore_testok_restored_shard1_replica_p7",           
"core_node_name":"core_node8",           
"base_url":"http://127.0.0.1:57671/solr";,           
"node_name":"127.0.0.1:57671_solr",           "state":"active",           
"collection":"backuprestore_testok_restored",           "shard":"shard1",       
    "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156444717626525"}},   "router":{"name":"compositeId"},   
"nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:57671_solr has 4 replicas. Expected 
num replicas : 2. state: 
DocCollection(backuprestore_testok_restored/17)={
  "pullReplicas":1,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"backuprestore_testok_restored_shard2_replica_n1",
          "leader":"true",
          "core_node_name":"core_node2",
          "base_url":"http://127.0.0.1:57671/solr";,
          "node_name":"127.0.0.1:57671_solr",
          "state":"active",
          "collection":"backuprestore_testok_restored",
          "shard":"shard2",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node6":{
          "core":"backuprestore_testok_restored_shard2_replica_p5",
          "core_node_name":"core_node6",
          "base_url":"http://127.0.0.1:57671/solr";,
          "node_name":"127.0.0.1:57671_solr",
          "state":"active",
          "collection":"backuprestore_testok_restored",
          "shard":"shard2",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156444717550765"},
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"backuprestore_testok_restored_shard1_replica_n3",
          "leader":"true",
          "core_node_name":"core_node4",
          "base_url":"http://127.0.0.1:57671/solr";,
          "node_name":"127.0.0.1:57671_solr",
          "state":"active",
          "collection":"backuprestore_testok_restored",
          "shard":"shard1",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node8":{
          "core":"backuprestore_testok_restored_shard1_replica_p7",
          "core_node_name":"core_node8",
          "base_url":"http://127.0.0.1:57671/solr";,
          "node_name":"127.0.0.1:57671_solr",
          "state":"active",
          "collection":"backuprestore_testok_restored",
          "shard":"shard1",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156444717626525"}},
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0}
        at 
__randomizedtesting.SeedInfo.seed([B4A5292B78D40EF3:3CF116F1D628630B]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:374)
        at java.base/java.util.HashMap.forEach(HashMap.java:1336)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:373)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:155)
        at 
org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:93)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
        at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  
org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Node 127.0.0.1:57684_solr has 4 replicas. Expected num replicas : 2. state:  
DocCollection(backuprestore_testok_restored/16)={   "pullReplicas":1,   
"replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",  
     "state":"active",       "replicas":{         "core_node2":{           
"core":"backuprestore_testok_restored_shard2_replica_n1",           
"leader":"true",           "core_node_name":"core_node2",           
"base_url":"http://127.0.0.1:57684/solr";,           
"node_name":"127.0.0.1:57684_solr",           "state":"active",           
"collection":"backuprestore_testok_restored",           "shard":"shard2",       
    "type":"NRT",           "force_set_state":"false"},         "core_node6":{  
         "core":"backuprestore_testok_restored_shard2_replica_p5",           
"core_node_name":"core_node6",           
"base_url":"http://127.0.0.1:57684/solr";,           
"node_name":"127.0.0.1:57684_solr",           "state":"active",           
"collection":"backuprestore_testok_restored",           "shard":"shard2",       
    "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156447856911615"},     "shard1":{       
"range":"80000000-ffffffff",       "state":"active",       "replicas":{         
"core_node4":{           
"core":"backuprestore_testok_restored_shard1_replica_n3",           
"leader":"true",           "core_node_name":"core_node4",           
"base_url":"http://127.0.0.1:57684/solr";,           
"node_name":"127.0.0.1:57684_solr",           "state":"active",           
"collection":"backuprestore_testok_restored",           "shard":"shard1",       
    "type":"NRT",           "force_set_state":"false"},         "core_node8":{  
         "core":"backuprestore_testok_restored_shard1_replica_p7",           
"core_node_name":"core_node8",           
"base_url":"http://127.0.0.1:57684/solr";,           
"node_name":"127.0.0.1:57684_solr",           "state":"active",           
"collection":"backuprestore_testok_restored",           "shard":"shard1",       
    "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156447856945446"}},   "router":{"name":"compositeId"},   
"nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:57684_solr has 4 replicas. Expected 
num replicas : 2. state: 
DocCollection(backuprestore_testok_restored/16)={
  "pullReplicas":1,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"backuprestore_testok_restored_shard2_replica_n1",
          "leader":"true",
          "core_node_name":"core_node2",
          "base_url":"http://127.0.0.1:57684/solr";,
          "node_name":"127.0.0.1:57684_solr",
          "state":"active",
          "collection":"backuprestore_testok_restored",
          "shard":"shard2",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node6":{
          "core":"backuprestore_testok_restored_shard2_replica_p5",
          "core_node_name":"core_node6",
          "base_url":"http://127.0.0.1:57684/solr";,
          "node_name":"127.0.0.1:57684_solr",
          "state":"active",
          "collection":"backuprestore_testok_restored",
          "shard":"shard2",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156447856911615"},
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"backuprestore_testok_restored_shard1_replica_n3",
          "leader":"true",
          "core_node_name":"core_node4",
          "base_url":"http://127.0.0.1:57684/solr";,
          "node_name":"127.0.0.1:57684_solr",
          "state":"active",
          "collection":"backuprestore_testok_restored",
          "shard":"shard1",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node8":{
          "core":"backuprestore_testok_restored_shard1_replica_p7",
          "core_node_name":"core_node8",
          "base_url":"http://127.0.0.1:57684/solr";,
          "node_name":"127.0.0.1:57684_solr",
          "state":"active",
          "collection":"backuprestore_testok_restored",
          "shard":"shard1",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156447856945446"}},
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0}
        at 
__randomizedtesting.SeedInfo.seed([B4A5292B78D40EF3:3CF116F1D628630B]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:374)
        at java.base/java.util.HashMap.forEach(HashMap.java:1336)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:373)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:155)
        at 
org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:93)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
        at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  
org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Node 127.0.0.1:57698_solr has 4 replicas. Expected num replicas : 2. state:  
DocCollection(backuprestore_testok_restored/15)={   "pullReplicas":1,   
"replicationFactor":1,   "shards":{     "shard2":{       "range":"0-7fffffff",  
     "state":"active",       "replicas":{         "core_node2":{           
"core":"backuprestore_testok_restored_shard2_replica_n1",           
"leader":"true",           "core_node_name":"core_node2",           
"base_url":"http://127.0.0.1:57698/solr";,           
"node_name":"127.0.0.1:57698_solr",           "state":"active",           
"collection":"backuprestore_testok_restored",           "shard":"shard2",       
    "type":"NRT",           "force_set_state":"false"},         "core_node6":{  
         "core":"backuprestore_testok_restored_shard2_replica_p5",           
"core_node_name":"core_node6",           
"base_url":"http://127.0.0.1:57698/solr";,           
"node_name":"127.0.0.1:57698_solr",           "state":"active",           
"collection":"backuprestore_testok_restored",           "shard":"shard2",       
    "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156449695846216"},     "shard1":{       
"range":"80000000-ffffffff",       "state":"active",       "replicas":{         
"core_node4":{           
"core":"backuprestore_testok_restored_shard1_replica_n3",           
"leader":"true",           "core_node_name":"core_node4",           
"base_url":"http://127.0.0.1:57698/solr";,           
"node_name":"127.0.0.1:57698_solr",           "state":"active",           
"collection":"backuprestore_testok_restored",           "shard":"shard1",       
    "type":"NRT",           "force_set_state":"false"},         "core_node8":{  
         "core":"backuprestore_testok_restored_shard1_replica_p7",           
"core_node_name":"core_node8",           
"base_url":"http://127.0.0.1:57698/solr";,           
"node_name":"127.0.0.1:57698_solr",           "state":"active",           
"collection":"backuprestore_testok_restored",           "shard":"shard1",       
    "type":"PULL",           "force_set_state":"false"}},       
"stateTimestamp":"1597156449695895085"}},   "router":{"name":"compositeId"},   
"nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:57698_solr has 4 replicas. Expected 
num replicas : 2. state: 
DocCollection(backuprestore_testok_restored/15)={
  "pullReplicas":1,
  "replicationFactor":1,
  "shards":{
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "core":"backuprestore_testok_restored_shard2_replica_n1",
          "leader":"true",
          "core_node_name":"core_node2",
          "base_url":"http://127.0.0.1:57698/solr";,
          "node_name":"127.0.0.1:57698_solr",
          "state":"active",
          "collection":"backuprestore_testok_restored",
          "shard":"shard2",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node6":{
          "core":"backuprestore_testok_restored_shard2_replica_p5",
          "core_node_name":"core_node6",
          "base_url":"http://127.0.0.1:57698/solr";,
          "node_name":"127.0.0.1:57698_solr",
          "state":"active",
          "collection":"backuprestore_testok_restored",
          "shard":"shard2",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156449695846216"},
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"backuprestore_testok_restored_shard1_replica_n3",
          "leader":"true",
          "core_node_name":"core_node4",
          "base_url":"http://127.0.0.1:57698/solr";,
          "node_name":"127.0.0.1:57698_solr",
          "state":"active",
          "collection":"backuprestore_testok_restored",
          "shard":"shard1",
          "type":"NRT",
          "force_set_state":"false"},
        "core_node8":{
          "core":"backuprestore_testok_restored_shard1_replica_p7",
          "core_node_name":"core_node8",
          "base_url":"http://127.0.0.1:57698/solr";,
          "node_name":"127.0.0.1:57698_solr",
          "state":"active",
          "collection":"backuprestore_testok_restored",
          "shard":"shard1",
          "type":"PULL",
          "force_set_state":"false"}},
      "stateTimestamp":"1597156449695895085"}},
  "router":{"name":"compositeId"},
  "nrtReplicas":1,
  "tlogReplicas":0}
        at 
__randomizedtesting.SeedInfo.seed([B4A5292B78D40EF3:3CF116F1D628630B]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:374)
        at java.base/java.util.HashMap.forEach(HashMap.java:1336)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:373)
        at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:155)
        at 
org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:93)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.ev

[...truncated too long message...]

a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty 
tag=null
   [junit4]   2> 252111 INFO  (jetty-closer-594-thread-1) [     ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@78ac4039: rootName = 
solr_57721, domain = solr.jetty, service url = null, agent id = null] for 
registry solr.jetty/com.codahale.metrics.MetricRegistry@14e45573
   [junit4]   2> 252111 INFO  (jetty-closer-594-thread-1) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster 
tag=null
   [junit4]   2> 252119 INFO  (closeThreadPool-606-thread-2) [     ] 
o.a.s.c.Overseer Overseer 
(id=72181706035822597-127.0.0.1:57721_solr-n_0000000000) closing
   [junit4]   2> 252119 INFO  
(OverseerStateUpdate-72181706035822597-127.0.0.1:57721_solr-n_0000000000) 
[n:127.0.0.1:57721_solr     ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:57721_solr
   [junit4]   2> 252169 INFO  (closeThreadPool-606-thread-1) [     ] 
o.a.s.c.Overseer Overseer 
(id=72181706035822597-127.0.0.1:57721_solr-n_0000000000) closing
   [junit4]   2> 252160 WARN  
(jetty-launcher-461-thread-1-SendThread(127.0.0.1:57713)) [     ] 
o.a.z.ClientCnxn An exception was thrown while closing send thread for session 
0x10070e1138c0005.
   [junit4]   2>           => EndOfStreamException: Unable to read additional 
data from server sessionid 0x10070e1138c0005, likely server has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
   [junit4]   2> org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable 
to read additional data from server sessionid 0x10070e1138c0005, likely server 
has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75) 
~[zookeeper-3.6.1.jar:3.6.1]
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348)
 ~[zookeeper-3.6.1.jar:3.6.1]
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262) 
[zookeeper-3.6.1.jar:3.6.1]
   [junit4]   2> 252189 INFO  (jetty-closer-594-thread-2) [     ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@6f7ac003{HTTP/1.1, (http/1.1, 
h2c)}{127.0.0.1:0}
   [junit4]   2> 252192 INFO  (jetty-closer-594-thread-2) [     ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@2ce31a6c{/solr,null,UNAVAILABLE}
   [junit4]   2> 252192 INFO  (jetty-closer-594-thread-2) [     ] 
o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 252293 INFO  (jetty-closer-594-thread-1) [     ] 
o.a.s.c.Overseer Overseer 
(id=72181706035822597-127.0.0.1:57721_solr-n_0000000000) closing
   [junit4]   2> 252296 INFO  (jetty-closer-594-thread-1) [     ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@17480a9b{HTTP/1.1, (http/1.1, 
h2c)}{127.0.0.1:0}
   [junit4]   2> 252297 INFO  (jetty-closer-594-thread-1) [     ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@33fe17c8{/solr,null,UNAVAILABLE}
   [junit4]   2> 252297 INFO  (jetty-closer-594-thread-1) [     ] 
o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 252299 INFO  
(SUITE-TestLocalFSCloudBackupRestore-seed#[B4A5292B78D40EF3]-worker) [     ] 
o.a.s.c.ZkTestServer Shutting down ZkTestServer.
   [junit4]   2> 252414 WARN  
(SUITE-TestLocalFSCloudBackupRestore-seed#[B4A5292B78D40EF3]-worker-SendThread(127.0.0.1:57713))
 [     ] o.a.z.ClientCnxn An exception was thrown while closing send thread for 
session 0x10070e1138c0001.
   [junit4]   2>           => EndOfStreamException: Unable to read additional 
data from server sessionid 0x10070e1138c0001, likely server has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
   [junit4]   2> org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable 
to read additional data from server sessionid 0x10070e1138c0001, likely server 
has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75) 
~[zookeeper-3.6.1.jar:3.6.1]
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348)
 ~[zookeeper-3.6.1.jar:3.6.1]
   [junit4]   2>        at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262) 
[zookeeper-3.6.1.jar:3.6.1]
   [junit4]   2> 252539 WARN  (ZkTestServer Run Thread) [     ] 
o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        7       
/solr/collections/backuprestore_testfailure/terms/shard1
   [junit4]   2>        7       
/solr/collections/backuprestore_testfailure/terms/shard2
   [junit4]   2>        5       
/solr/collections/backuprestore_testok/terms/shard1
   [junit4]   2>        5       
/solr/collections/backuprestore_testok/terms/shard2
   [junit4]   2>        3       /solr/clusterprops.json
   [junit4]   2>        3       
/solr/collections/backuprestore_testok_restored/terms/shard2
   [junit4]   2>        3       
/solr/collections/backuprestore_testok_restored/terms/shard1
   [junit4]   2>        3       /solr/aliases.json
   [junit4]   2>        2       
/solr/collections/backuprestore_testfailure/collectionprops.json
   [junit4]   2>        2       /solr/packages.json
   [junit4]   2>        2       /solr/security.json
   [junit4]   2>        2       /solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        46      
/solr/collections/backuprestore_testok_restored/state.json
   [junit4]   2>        36      
/solr/collections/backuprestore_testok/state.json
   [junit4]   2>        29      
/solr/collections/backuprestore_testfailure/state.json
   [junit4]   2>        3       
/solr/collections/backuprestore_testfailure_restored/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        90      /solr/overseer/queue
   [junit4]   2>        18      /solr/collections
   [junit4]   2>        18      /solr/overseer/collection-queue-work
   [junit4]   2>        8       /solr/live_nodes
   [junit4]   2> 
   [junit4]   2> 252556 INFO  
(SUITE-TestLocalFSCloudBackupRestore-seed#[B4A5292B78D40EF3]-worker) [     ] 
o.a.s.c.ZkTestServer waitForServerDown: 127.0.0.1:57713
   [junit4]   2> 252556 INFO  
(SUITE-TestLocalFSCloudBackupRestore-seed#[B4A5292B78D40EF3]-worker) [     ] 
o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:57713
   [junit4]   2> 252556 INFO  
(SUITE-TestLocalFSCloudBackupRestore-seed#[B4A5292B78D40EF3]-worker) [     ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1 57713
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.api.collections.TestLocalFSCloudBackupRestore_B4A5292B78D40EF3-001
   [junit4]   2> Aug 11, 2020 2:34:18 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=CheapBastard, 
sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=wo, timezone=GMT0
   [junit4]   2> NOTE: Mac OS X 10.14.6 x86_64/AdoptOpenJDK 11.0.6 
(64-bit)/cpus=6,threads=1,free=147757040,total=350224384
   [junit4]   2> NOTE: All tests run in this JVM: [TestHdfsCloudBackupRestore, 
TestBulkSchemaConcurrent, TestLocalFSCloudBackupRestore]
   [junit4] Completed [15/15 (11!)] on J0 in 33.39s, 2 tests, 1 failure <<< 
FAILURES!

[...truncated 22 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1521:
 The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1048:
 There were test failures: 15 suites, 25 tests, 2 suite-level errors, 1 error, 
10 failures [seed: B4A5292B78D40EF3]

Total time: 4 minutes 15 seconds

[repro] Setting last failure code to 256

[repro] Failures w/original seeds:
[repro]   1/5 failed: org.apache.solr.schema.TestBulkSchemaConcurrent
[repro]   5/5 failed: 
org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore
[repro]   5/5 failed: 
org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Archiving artifacts
[Java] Sleeping for 5 seconds due to JENKINS-32191...
[Java] Parsing console log (workspace: 
'/Users/jenkins/workspace/Lucene-Solr-master-MacOSX')
[Java] Successfully parsed console log
[Java] -> found 0 issues (skipped 0 duplicates)
[Java] Skipping post processing
[Java] No filter has been set, publishing all 0 issues
[Java] Using reference build 'Lucene-Solr-master-MacOSX #6402' to compute new, 
fixed, and outstanding issues
[Java] Issues delta (vs. reference build): outstanding: 0, new: 0, fixed: 0
[Java] No quality gates have been set - skipping
[Java] Health report is disabled - skipping
[Java] Created analysis result for 0 issues (found 0 new issues, fixed 0 issues)
[Java] Attaching ResultAction with ID 'java' to run 'Lucene-Solr-master-MacOSX 
#6403'.
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Recording test results
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to