Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/198/

4 tests failed.
FAILED:  
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testResilienceWithDeleteByQueryOnTarget

Error Message:
Timeout while trying to assert number of documents @ target_collection

Stack Trace:
java.lang.AssertionError: Timeout while trying to assert number of documents @ 
target_collection
        at 
__randomizedtesting.SeedInfo.seed([B6A88B4914AF140A:16CA19D3C4F6D14F]:0)
        at 
org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:271)
        at 
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testResilienceWithDeleteByQueryOnTarget(CdcrReplicationDistributedZkTest.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: expected:<50> but was:<0>
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at 
org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:262)
        ... 42 more


FAILED:  
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1479074097480,generation=2,filelist=[_ml.cfe, _ml.cfs, 
_ml.si, _mm.cfe, _mm.cfs, _mm.si, _mo.fdt, _mo.fdx, _mo.fnm, _mo.nvd, _mo.nvm, 
_mo.si, _mo_FSTOrd50_0.doc, _mo_FSTOrd50_0.tbk, _mo_FSTOrd50_0.tix, _mp.cfe, 
_mp.cfs, _mp.si, _mr.cfe, _mr.cfs, _mr.si, segments_2]}]> but 
was:<[{indexVersion=1479074097480,generation=2,filelist=[_ml.cfe, _ml.cfs, 
_ml.si, _mm.cfe, _mm.cfs, _mm.si, _mo.fdt, _mo.fdx, _mo.fnm, _mo.nvd, _mo.nvm, 
_mo.si, _mo_FSTOrd50_0.doc, _mo_FSTOrd50_0.tbk, _mo_FSTOrd50_0.tix, _mp.cfe, 
_mp.cfs, _mp.si, _mr.cfe, _mr.cfs, _mr.si, segments_2]}, 
{indexVersion=1479074097480,generation=3,filelist=[_mp.cfe, _mp.cfs, _mp.si, 
_mq.cfe, _mq.cfs, _mq.si, _mr.cfe, _mr.cfs, _mr.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: 
expected:<[{indexVersion=1479074097480,generation=2,filelist=[_ml.cfe, _ml.cfs, 
_ml.si, _mm.cfe, _mm.cfs, _mm.si, _mo.fdt, _mo.fdx, _mo.fnm, _mo.nvd, _mo.nvm, 
_mo.si, _mo_FSTOrd50_0.doc, _mo_FSTOrd50_0.tbk, _mo_FSTOrd50_0.tix, _mp.cfe, 
_mp.cfs, _mp.si, _mr.cfe, _mr.cfs, _mr.si, segments_2]}]> but 
was:<[{indexVersion=1479074097480,generation=2,filelist=[_ml.cfe, _ml.cfs, 
_ml.si, _mm.cfe, _mm.cfs, _mm.si, _mo.fdt, _mo.fdx, _mo.fnm, _mo.nvd, _mo.nvm, 
_mo.si, _mo_FSTOrd50_0.doc, _mo_FSTOrd50_0.tbk, _mo_FSTOrd50_0.tix, _mp.cfe, 
_mp.cfs, _mp.si, _mr.cfe, _mr.cfs, _mr.si, segments_2]}, 
{indexVersion=1479074097480,generation=3,filelist=[_mp.cfe, _mp.cfs, _mp.si, 
_mq.cfe, _mq.cfs, _mq.si, _mr.cfe, _mr.cfs, _mr.si, segments_3]}]>
        at 
__randomizedtesting.SeedInfo.seed([B6A88B4914AF140A:937F907964E71A09]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at 
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1229)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testSpecificConfigsets

Error Message:
KeeperErrorCode = NoNode for /collections/withconfigset2

Stack Trace:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode 
for /collections/withconfigset2
        at 
__randomizedtesting.SeedInfo.seed([B6A88B4914AF140A:9BD6C413E38BAE06]:0)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:356)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:353)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testSpecificConfigsets(CollectionsAPIDistributedZkTest.java:425)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testCollectionsAPI

Error Message:
Expected to see collection awhollynewcollection_0 null Last available state: 
DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/5)={
   "replicationFactor":"1",   "shards":{     "shard1":{       
"range":"80000000-bfffffff",       "state":"active",       "replicas":{}},     
"shard2":{       "range":"c0000000-ffffffff",       "state":"active",       
"replicas":{}},     "shard3":{       "range":"0-3fffffff",       
"state":"active",       "replicas":{}},     "shard4":{       
"range":"40000000-7fffffff",       "state":"active",       "replicas":{}}},   
"router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected to see collection awhollynewcollection_0
null
Last available state: 
DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/5)={
  "replicationFactor":"1",
  "shards":{
    "shard1":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{}},
    "shard2":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{}},
    "shard3":{
      "range":"0-3fffffff",
      "state":"active",
      "replicas":{}},
    "shard4":{
      "range":"40000000-7fffffff",
      "state":"active",
      "replicas":{}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false"}
        at 
__randomizedtesting.SeedInfo.seed([B6A88B4914AF140A:FEDDFFFD129C3B9F]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:236)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:496)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11514 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_B6A88B4914AF140A-001/init-core-data-001
   [junit4]   2> 740332 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 740333 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_B6A88B4914AF140A-001/tempDir-001
   [junit4]   2> 740333 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 740334 INFO  (Thread-5110) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 740334 INFO  (Thread-5110) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 740434 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.a.s.c.ZkTestServer start zk server on port:60131
   [junit4]   2> 740456 INFO  (jetty-launcher-701-thread-1) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 740458 INFO  (jetty-launcher-701-thread-2) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 740460 INFO  (jetty-launcher-701-thread-3) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 740462 INFO  (jetty-launcher-701-thread-4) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 740464 INFO  (jetty-launcher-701-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@6f0de7ec{/solr,null,AVAILABLE}
   [junit4]   2> 740465 INFO  (jetty-launcher-701-thread-2) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@521e773d{SSL,[ssl, 
http/1.1]}{127.0.0.1:40417}
   [junit4]   2> 740465 INFO  (jetty-launcher-701-thread-2) [    ] 
o.e.j.s.Server Started @744433ms
   [junit4]   2> 740466 INFO  (jetty-launcher-701-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=40417}
   [junit4]   2> 740466 ERROR (jetty-launcher-701-thread-2) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 740477 INFO  (jetty-launcher-701-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 740477 INFO  (jetty-launcher-701-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 740477 INFO  (jetty-launcher-701-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 740477 INFO  (jetty-launcher-701-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-11-13T21:16:40.732Z
   [junit4]   2> 740470 INFO  (jetty-launcher-701-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@e82bc48{/solr,null,AVAILABLE}
   [junit4]   2> 740469 INFO  (jetty-launcher-701-thread-4) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@753ba8ad{/solr,null,AVAILABLE}
   [junit4]   2> 740467 INFO  (jetty-launcher-701-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@49edbafa{/solr,null,AVAILABLE}
   [junit4]   2> 740487 INFO  (jetty-launcher-701-thread-1) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@36bd0e10{SSL,[ssl, 
http/1.1]}{127.0.0.1:47633}
   [junit4]   2> 740487 INFO  (jetty-launcher-701-thread-1) [    ] 
o.e.j.s.Server Started @744455ms
   [junit4]   2> 740487 INFO  (jetty-launcher-701-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=47633}
   [junit4]   2> 740488 ERROR (jetty-launcher-701-thread-1) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 740488 INFO  (jetty-launcher-701-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 740488 INFO  (jetty-launcher-701-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 740488 INFO  (jetty-launcher-701-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 740488 INFO  (jetty-launcher-701-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-11-13T21:16:40.743Z
   [junit4]   2> 740490 INFO  (jetty-launcher-701-thread-4) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@1856dc7c{SSL,[ssl, 
http/1.1]}{127.0.0.1:39538}
   [junit4]   2> 740490 INFO  (jetty-launcher-701-thread-4) [    ] 
o.e.j.s.Server Started @744457ms
   [junit4]   2> 740490 INFO  (jetty-launcher-701-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=39538}
   [junit4]   2> 740490 ERROR (jetty-launcher-701-thread-4) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 740490 INFO  (jetty-launcher-701-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 740490 INFO  (jetty-launcher-701-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 740490 INFO  (jetty-launcher-701-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 740490 INFO  (jetty-launcher-701-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-11-13T21:16:40.745Z
   [junit4]   2> 740492 INFO  (jetty-launcher-701-thread-3) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@1ee46472{SSL,[ssl, 
http/1.1]}{127.0.0.1:60969}
   [junit4]   2> 740492 INFO  (jetty-launcher-701-thread-3) [    ] 
o.e.j.s.Server Started @744460ms
   [junit4]   2> 740492 INFO  (jetty-launcher-701-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=60969}
   [junit4]   2> 740493 ERROR (jetty-launcher-701-thread-3) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 740493 INFO  (jetty-launcher-701-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 740493 INFO  (jetty-launcher-701-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 740493 INFO  (jetty-launcher-701-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 740493 INFO  (jetty-launcher-701-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-11-13T21:16:40.748Z
   [junit4]   2> 740499 INFO  (jetty-launcher-701-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 740502 INFO  (jetty-launcher-701-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 740506 INFO  (jetty-launcher-701-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 740512 INFO  (jetty-launcher-701-thread-1) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 740513 INFO  (jetty-launcher-701-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 740513 INFO  (jetty-launcher-701-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60131/solr
   [junit4]   2> 740524 INFO  (jetty-launcher-701-thread-4) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 740526 INFO  (jetty-launcher-701-thread-2) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 740529 INFO  (jetty-launcher-701-thread-3) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 740537 INFO  (jetty-launcher-701-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60131/solr
   [junit4]   2> 740538 INFO  (jetty-launcher-701-thread-1) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:47633_solr
   [junit4]   2> 740557 INFO  (jetty-launcher-701-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60131/solr
   [junit4]   2> 740557 INFO  (jetty-launcher-701-thread-4) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60131/solr
   [junit4]   2> 740571 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x1585f8c9d0e0007, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 740573 INFO  (jetty-launcher-701-thread-1) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.Overseer Overseer 
(id=96932449523466246-127.0.0.1:47633_solr-n_0000000000) starting
   [junit4]   2> 740602 INFO  (jetty-launcher-701-thread-3) 
[n:127.0.0.1:60969_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:60969_solr
   [junit4]   2> 740602 INFO  (jetty-launcher-701-thread-4) 
[n:127.0.0.1:39538_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:39538_solr
   [junit4]   2> 740614 INFO  (jetty-launcher-701-thread-1) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:47633_solr
   [junit4]   2> 740619 INFO  
(zkCallback-710-thread-1-processing-n:127.0.0.1:47633_solr) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 740621 INFO  
(zkCallback-722-thread-2-processing-n:127.0.0.1:39538_solr) 
[n:127.0.0.1:39538_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 740626 INFO  (jetty-launcher-701-thread-2) 
[n:127.0.0.1:40417_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (3)
   [junit4]   2> 740629 INFO  
(OverseerStateUpdate-96932449523466246-127.0.0.1:47633_solr-n_0000000000) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 740633 INFO  (jetty-launcher-701-thread-4) 
[n:127.0.0.1:39538_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_B6A88B4914AF140A-001/tempDir-001/node4/.
   [junit4]   2> 740636 INFO  
(zkCallback-721-thread-1-processing-n:127.0.0.1:60969_solr) 
[n:127.0.0.1:60969_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 740636 INFO  (jetty-launcher-701-thread-3) 
[n:127.0.0.1:60969_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_B6A88B4914AF140A-001/tempDir-001/node3/.
   [junit4]   2> 740637 INFO  
(zkCallback-722-thread-1-processing-n:127.0.0.1:39538_solr) 
[n:127.0.0.1:39538_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 740638 INFO  
(zkCallback-721-thread-2-processing-n:127.0.0.1:60969_solr) 
[n:127.0.0.1:60969_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 740639 INFO  (jetty-launcher-701-thread-2) 
[n:127.0.0.1:40417_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:40417_solr
   [junit4]   2> 740640 INFO  
(zkCallback-722-thread-1-processing-n:127.0.0.1:39538_solr) 
[n:127.0.0.1:39538_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 740646 INFO  
(zkCallback-721-thread-2-processing-n:127.0.0.1:60969_solr) 
[n:127.0.0.1:60969_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 740667 INFO  
(zkCallback-723-thread-1-processing-n:127.0.0.1:40417_solr) 
[n:127.0.0.1:40417_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 740667 INFO  (jetty-launcher-701-thread-1) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_B6A88B4914AF140A-001/tempDir-001/node1/.
   [junit4]   2> 740675 INFO  
(zkCallback-710-thread-2-processing-n:127.0.0.1:47633_solr) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 740678 INFO  (jetty-launcher-701-thread-2) 
[n:127.0.0.1:40417_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_B6A88B4914AF140A-001/tempDir-001/node2/.
   [junit4]   2> 740684 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 740731 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 740745 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 740747 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 740760 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_59224_hdfs____fh0aka/webapp
   [junit4]   2> 741150 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59224
   [junit4]   2> 741214 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 741215 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 741228 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_53022_datanode____.g8hipz/webapp
   [junit4]   2> 741647 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53022
   [junit4]   2> 741680 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 741681 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 741697 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_39851_datanode____.3gndrf/webapp
   [junit4]   2> 741762 INFO  (IPC Server handler 4 on 56264) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-c7e0169e-71bd-4ec1-8526-d3277a03a491 node 
DatanodeRegistration(127.0.0.1:43634, 
datanodeUuid=837f1d5f-a258-4e15-b631-8abc100e09aa, infoPort=44172, 
infoSecurePort=0, ipcPort=35246, 
storageInfo=lv=-56;cid=testClusterID;nsid=1100650242;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 741762 INFO  (IPC Server handler 4 on 56264) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-55ad8a82-6086-42a1-b25e-86b98b667462 node 
DatanodeRegistration(127.0.0.1:43634, 
datanodeUuid=837f1d5f-a258-4e15-b631-8abc100e09aa, infoPort=44172, 
infoSecurePort=0, ipcPort=35246, 
storageInfo=lv=-56;cid=testClusterID;nsid=1100650242;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 742141 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39851
   [junit4]   2> 742283 INFO  (IPC Server handler 3 on 56264) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-cf78a4ba-600b-4c9e-8d1d-5f2080d81c7d node 
DatanodeRegistration(127.0.0.1:57085, 
datanodeUuid=d007fb96-3a84-428b-94a7-d966f367e2fa, infoPort=34140, 
infoSecurePort=0, ipcPort=51957, 
storageInfo=lv=-56;cid=testClusterID;nsid=1100650242;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 742283 INFO  (IPC Server handler 3 on 56264) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-dce15cc9-5144-4fc7-b359-a7279b3a1f71 node 
DatanodeRegistration(127.0.0.1:57085, 
datanodeUuid=d007fb96-3a84-428b-94a7-d966f367e2fa, infoPort=34140, 
infoSecurePort=0, ipcPort=51957, 
storageInfo=lv=-56;cid=testClusterID;nsid=1100650242;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 742378 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[B6A88B4914AF140A])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testTooManyReplicas
   [junit4]   2> 742380 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[B6A88B4914AF140A])
 [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 742442 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=10&collection.configName=conf&name=collection&action=CREATE&numShards=2&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 742445 INFO  
(OverseerThreadFactory-2132-thread-1-processing-n:127.0.0.1:47633_solr) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
collection
   [junit4]   2> 742446 WARN  
(OverseerThreadFactory-2132-thread-1-processing-n:127.0.0.1:47633_solr) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.CreateCollectionCmd Specified 
replicationFactor of 10 on collection collection is higher than or equal to the 
number of Solr instances currently live or live and part of your 
createNodeSet(4). It's unusual to run two replica of the same slice on the same 
Solr-instance.
   [junit4]   2> 742446 ERROR 
(OverseerThreadFactory-2132-thread-1-processing-n:127.0.0.1:47633_solr) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.OverseerCollectionMessageHandler 
Collection: collection operation: create 
failed:org.apache.solr.common.SolrException: Cannot create collection 
collection. Value of maxShardsPerNode is 1, and the number of nodes currently 
live or live and part of your createNodeSet is 4. This allows a maximum of 4 to 
be created. Value of numShards is 2 and value of replicationFactor is 10. This 
requires 20 shards to be created (higher than the allowed number)
   [junit4]   2>        at 
org.apache.solr.cloud.CreateCollectionCmd.call(CreateCollectionCmd.java:148)
   [junit4]   2>        at 
org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:224)
   [junit4]   2>        at 
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:463)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 742450 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=10&collection.configName=conf&name=collection&action=CREATE&numShards=2&wt=javabin&version=2}
 status=400 QTime=7
   [junit4]   2> 742452 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[B6A88B4914AF140A])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testTooManyReplicas
   [junit4]   2> 742522 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes-seed#[B6A88B4914AF140A])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testCoresAreDistributedAcrossNodes
   [junit4]   2> 742523 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes-seed#[B6A88B4914AF140A])
 [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 742526 INFO  (qtp1955811460-8826) [n:127.0.0.1:47633_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&collection.configName=conf&name=nodes_used_collection&action=CREATE&numShards=2&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 742528 INFO  
(OverseerThreadFactory-2132-thread-2-processing-n:127.0.0.1:47633_solr) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
nodes_used_collection
   [junit4]   2> 742529 INFO  
(OverseerCollectionConfigSetProcessor-96932449523466246-127.0.0.1:47633_solr-n_0000000000)
 [n:127.0.0.1:47633_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 742690 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 742721 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 742728 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 742736 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 742840 INFO  
(zkCallback-722-thread-1-processing-n:127.0.0.1:39538_solr) 
[n:127.0.0.1:39538_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 742840 INFO  
(zkCallback-710-thread-4-processing-n:127.0.0.1:47633_solr) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 742840 INFO  
(zkCallback-723-thread-1-processing-n:127.0.0.1:40417_solr) 
[n:127.0.0.1:40417_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 742841 INFO  
(zkCallback-721-thread-2-processing-n:127.0.0.1:60969_solr) 
[n:127.0.0.1:60969_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 743706 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 743715 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica1] 
o.a.s.s.IndexSchema [nodes_used_collection_shard1_replica1] Schema name=minimal
   [junit4]   2> 743718 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica1] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 743719 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica1] 
o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard1_replica1' 
using configuration from collection nodes_used_collection
   [junit4]   2> 743727 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:56264/data
   [junit4]   2> 743727 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Solr 
Kerberos Authentication disabled
   [junit4]   2> 743727 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.SolrCore 
[[nodes_used_collection_shard1_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_B6A88B4914AF140A-001/tempDir-001/node1/nodes_used_collection_shard1_replica1],
 dataDir=[hdfs://localhost:56264/data/nodes_used_collection/core_node1/data/]
   [junit4]   2> 743727 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:56264/data/nodes_used_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 743737 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 743737 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 743737 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Creating 
new single instance HDFS BlockCache
   [junit4]   2> 743744 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 743744 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 743749 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 743754 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.s.IndexSchema [nodes_used_collection_shard1_replica2] Schema name=minimal
   [junit4]   2> 743754 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.s.IndexSchema [nodes_used_collection_shard2_replica2] Schema name=minimal
   [junit4]   2> 743771 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 743773 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica1] 
o.a.s.s.IndexSchema [nodes_used_collection_shard2_replica1] Schema name=minimal
   [junit4]   2> 743773 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 743776 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard1_replica2' 
using configuration from collection nodes_used_collection
   [junit4]   2> 743779 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 743780 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:56264/data/nodes_used_collection/core_node1/data
   [junit4]   2> 743780 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica1] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 743781 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica1] 
o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard2_replica1' 
using configuration from collection nodes_used_collection
   [junit4]   2> 743781 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:56264/data
   [junit4]   2> 743782 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Solr 
Kerberos Authentication disabled
   [junit4]   2> 743782 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.SolrCore 
[[nodes_used_collection_shard2_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_B6A88B4914AF140A-001/tempDir-001/node2/nodes_used_collection_shard2_replica1],
 dataDir=[hdfs://localhost:56264/data/nodes_used_collection/core_node4/data/]
   [junit4]   2> 743782 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:56264/data/nodes_used_collection/core_node4/data/snapshot_metadata
   [junit4]   2> 743786 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:56264/data
   [junit4]   2> 743786 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Solr 
Kerberos Authentication disabled
   [junit4]   2> 743786 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.SolrCore 
[[nodes_used_collection_shard1_replica2] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_B6A88B4914AF140A-001/tempDir-001/node4/nodes_used_collection_shard1_replica2],
 dataDir=[hdfs://localhost:56264/data/nodes_used_collection/core_node3/data/]
   [junit4]   2> 743786 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:56264/data/nodes_used_collection/core_node3/data/snapshot_metadata
   [junit4]   2> 743788 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 743788 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 743788 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Creating 
new single instance HDFS BlockCache
   [junit4]   2> 743778 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard2_replica2' 
using configuration from collection nodes_used_collection
   [junit4]   2> 743790 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:56264/data
   [junit4]   2> 743790 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Solr 
Kerberos Authentication disabled
   [junit4]   2> 743790 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.SolrCore 
[[nodes_used_collection_shard2_replica2] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_B6A88B4914AF140A-001/tempDir-001/node3/nodes_used_collection_shard2_replica2],
 dataDir=[hdfs://localhost:56264/data/nodes_used_collection/core_node2/data/]
   [junit4]   2> 743792 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:56264/data/nodes_used_collection/core_node2/data/snapshot_metadata
   [junit4]   2> 743797 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 743798 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 743798 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 743798 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Creating 
new single instance HDFS BlockCache
   [junit4]   2> 743803 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:56264/data/nodes_used_collection/core_node4/data
   [junit4]   2> 743805 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:56264/data/nodes_used_collection/core_node1/data/index
   [junit4]   2> 743810 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 743811 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:56264/data/nodes_used_collection/core_node3/data
   [junit4]   2> 743813 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 743813 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 743813 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Creating 
new single instance HDFS BlockCache
   [junit4]   2> 743818 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 743819 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 743819 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Creating 
new single instance HDFS BlockCache
   [junit4]   2> 743827 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 743828 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:56264/data/nodes_used_collection/core_node2/data
   [junit4]   2> 743834 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 743835 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:56264/data/nodes_used_collection/core_node3/data/index
   [junit4]   2> 743837 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:56264/data/nodes_used_collection/core_node4/data/index
   [junit4]   2> 743843 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 743844 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 743844 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Creating 
new single instance HDFS BlockCache
   [junit4]   2> 743846 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 743846 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 743846 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Creating 
new single instance HDFS BlockCache
   [junit4]   2> 743852 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:56264/data/nodes_used_collection/core_node2/data/index
   [junit4]   2> 743871 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 743872 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 743872 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Creating 
new single instance HDFS BlockCache
   [junit4]   2> 743875 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 743875 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 743904 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 743908 INFO  (IPC Server handler 0 on 56264) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57085 is 
added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-c7e0169e-71bd-4ec1-8526-d3277a03a491:NORMAL:127.0.0.1:43634|RBW],
 
ReplicaUC[[DISK]DS-cf78a4ba-600b-4c9e-8d1d-5f2080d81c7d:NORMAL:127.0.0.1:57085|FINALIZED]]}
 size 0
   [junit4]   2> 743909 INFO  (IPC Server handler 3 on 56264) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43634 is 
added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-cf78a4ba-600b-4c9e-8d1d-5f2080d81c7d:NORMAL:127.0.0.1:57085|FINALIZED],
 
ReplicaUC[[DISK]DS-55ad8a82-6086-42a1-b25e-86b98b667462:NORMAL:127.0.0.1:43634|FINALIZED]]}
 size 0
   [junit4]   2> 743917 INFO  (IPC Server handler 1 on 56264) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57085 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-55ad8a82-6086-42a1-b25e-86b98b667462:NORMAL:127.0.0.1:43634|RBW],
 
ReplicaUC[[DISK]DS-dce15cc9-5144-4fc7-b359-a7279b3a1f71:NORMAL:127.0.0.1:57085|FINALIZED]]}
 size 0
   [junit4]   2> 743919 INFO  (IPC Server handler 2 on 56264) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43634 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-dce15cc9-5144-4fc7-b359-a7279b3a1f71:NORMAL:127.0.0.1:57085|FINALIZED],
 
ReplicaUC[[DISK]DS-c7e0169e-71bd-4ec1-8526-d3277a03a491:NORMAL:127.0.0.1:43634|FINALIZED]]}
 size 0
   [junit4]   2> 743942 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 743942 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 743942 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.u.HdfsUpdateLog Initializing 
HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 743946 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 743946 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 743946 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.u.HdfsUpdateLog Initializing 
HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 743952 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 743952 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 743956 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 743956 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 743960 INFO  (IPC Server handler 1 on 56264) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43634 is 
added to blk_1073741828_1004{UCState=COMMITTED, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-55ad8a82-6086-42a1-b25e-86b98b667462:NORMAL:127.0.0.1:43634|RBW],
 
ReplicaUC[[DISK]DS-dce15cc9-5144-4fc7-b359-a7279b3a1f71:NORMAL:127.0.0.1:57085|RBW]]}
 size 71
   [junit4]   2> 743960 INFO  (IPC Server handler 9 on 56264) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57085 is 
added to blk_1073741828_1004 size 71
   [junit4]   2> 743961 INFO  (IPC Server handler 2 on 56264) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57085 is 
added to blk_1073741827_1003{UCState=COMMITTED, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-55ad8a82-6086-42a1-b25e-86b98b667462:NORMAL:127.0.0.1:43634|RBW],
 
ReplicaUC[[DISK]DS-cf78a4ba-600b-4c9e-8d1d-5f2080d81c7d:NORMAL:127.0.0.1:57085|RBW]]}
 size 71
   [junit4]   2> 743962 INFO  (IPC Server handler 3 on 56264) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43634 is 
added to blk_1073741827_1003 size 71
   [junit4]   2> 743971 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@15675385[nodes_used_collection_shard1_replica1] main]
   [junit4]   2> 743975 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 743975 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 743975 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.h.ReplicationHandler Commits 
will be reserved for  10000
   [junit4]   2> 743975 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1550919196192276480
   [junit4]   2> 743977 INFO  
(searcherExecutor-2141-thread-1-processing-n:127.0.0.1:47633_solr 
x:nodes_used_collection_shard1_replica1 s:shard1 c:nodes_used_collection 
r:core_node1) [n:127.0.0.1:47633_solr c:nodes_used_collection s:shard1 
r:core_node1 x:nodes_used_collection_shard1_replica1] o.a.s.c.SolrCore 
[nodes_used_collection_shard1_replica1] Registered new searcher 
Searcher@15675385[nodes_used_collection_shard1_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 743983 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext 
Waiting until we see more replicas up for shard shard1: total=2 found=1 
timeoutin=9999ms
   [junit4]   2> 743990 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@7b8066ab[nodes_used_collection_shard2_replica1] main]
   [junit4]   2> 743991 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 743992 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 743992 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.h.ReplicationHandler Commits 
will be reserved for  10000
   [junit4]   2> 743992 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1550919196210102272
   [junit4]   2> 743994 INFO  
(searcherExecutor-2142-thread-1-processing-n:127.0.0.1:40417_solr 
x:nodes_used_collection_shard2_replica1 s:shard2 c:nodes_used_collection 
r:core_node4) [n:127.0.0.1:40417_solr c:nodes_used_collection s:shard2 
r:core_node4 x:nodes_used_collection_shard2_replica1] o.a.s.c.SolrCore 
[nodes_used_collection_shard2_replica1] Registered new searcher 
Searcher@7b8066ab[nodes_used_collection_shard2_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 743999 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.ShardLeaderElectionContext 
Waiting until we see more replicas up for shard shard2: total=2 found=1 
timeoutin=9999ms
   [junit4]   2> 744102 INFO  
(zkCallback-722-thread-1-processing-n:127.0.0.1:39538_solr) 
[n:127.0.0.1:39538_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 744102 INFO  
(zkCallback-723-thread-1-processing-n:127.0.0.1:40417_solr) 
[n:127.0.0.1:40417_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 744102 INFO  
(zkCallback-721-thread-2-processing-n:127.0.0.1:60969_solr) 
[n:127.0.0.1:60969_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 744102 INFO  
(zkCallback-710-thread-1-processing-n:127.0.0.1:47633_solr) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 744374 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 744374 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 744374 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.u.HdfsUpdateLog Initializing 
HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 744379 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 744379 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 744379 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.u.HdfsUpdateLog Initializing 
HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 744383 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 744384 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 744392 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@5720ca5c[nodes_used_collection_shard2_replica2] main]
   [junit4]   2> 744393 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 744393 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 744394 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.h.ReplicationHandler Commits 
will be reserved for  10000
   [junit4]   2> 744394 INFO  (qtp50170523-8844) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1550919196631629824
   [junit4]   2> 744396 INFO  
(searcherExecutor-2144-thread-1-processing-n:127.0.0.1:60969_solr 
x:nodes_used_collection_shard2_replica2 s:shard2 c:nodes_used_collection 
r:core_node2) [n:127.0.0.1:60969_solr c:nodes_used_collection s:shard2 
r:core_node2 x:nodes_used_collection_shard2_replica2] o.a.s.c.SolrCore 
[nodes_used_collection_shard2_replica2] Registered new searcher 
Searcher@5720ca5c[nodes_used_collection_shard2_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 744400 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 744400 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 744413 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@8ebbd8c[nodes_used_collection_shard1_replica2] main]
   [junit4]   2> 744414 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 744414 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 744415 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.h.ReplicationHandler Commits 
will be reserved for  10000
   [junit4]   2> 744415 INFO  
(searcherExecutor-2143-thread-1-processing-n:127.0.0.1:39538_solr 
x:nodes_used_collection_shard1_replica2 s:shard1 c:nodes_used_collection 
r:core_node3) [n:127.0.0.1:39538_solr c:nodes_used_collection s:shard1 
r:core_node3 x:nodes_used_collection_shard1_replica2] o.a.s.c.SolrCore 
[nodes_used_collection_shard1_replica2] Registered new searcher 
Searcher@8ebbd8c[nodes_used_collection_shard1_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 744415 INFO  (qtp618199129-8834) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1550919196653649920
   [junit4]   2> 744483 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext 
Enough replicas found to continue.
   [junit4]   2> 744483 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I 
may be the new leader - try and sync
   [junit4]   2> 744483 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:47633/solr/nodes_used_collection_shard1_replica1/
   [junit4]   2> 744484 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.u.PeerSync PeerSync: 
core=nodes_used_collection_shard1_replica1 url=https://127.0.0.1:47633/solr 
START 
replicas=[https://127.0.0.1:39538/solr/nodes_used_collection_shard1_replica2/] 
nUpdates=100
   [junit4]   2> 744484 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.u.PeerSync PeerSync: 
core=nodes_used_collection_shard1_replica1 url=https://127.0.0.1:47633/solr 
DONE.  We have no versions.  sync failed.
   [junit4]   2> 744489 INFO  (qtp618199129-8832) [n:127.0.0.1:39538_solr 
c:nodes_used_collection s:shard1 r:core_node3 
x:nodes_used_collection_shard1_replica2] o.a.s.c.S.Request 
[nodes_used_collection_shard1_replica2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=0
   [junit4]   2> 744490 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.SyncStrategy Leader's attempt 
to sync with shard failed, moving to the next candidate
   [junit4]   2> 744490 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext We 
failed sync, but we have no versions - we can't sync in that case - we were 
active before, so become leader anyway
   [junit4]   2> 744494 INFO  (qtp1955811460-8825) [n:127.0.0.1:47633_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I 
am the new leader: 
https://127.0.0.1:47633/solr/nodes_used_collection_shard1_replica1/ shard1
   [junit4]   2> 744499 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.ShardLeaderElectionContext 
Enough replicas found to continue.
   [junit4]   2> 744499 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.ShardLeaderElectionContext I 
may be the new leader - try and sync
   [junit4]   2> 744499 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:40417/solr/nodes_used_collection_shard2_replica1/
   [junit4]   2> 744499 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.u.PeerSync PeerSync: 
core=nodes_used_collection_shard2_replica1 url=https://127.0.0.1:40417/solr 
START 
replicas=[https://127.0.0.1:60969/solr/nodes_used_collection_shard2_replica2/] 
nUpdates=100
   [junit4]   2> 744500 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.u.PeerSync PeerSync: 
core=nodes_used_collection_shard2_replica1 url=https://127.0.0.1:40417/solr 
DONE.  We have no versions.  sync failed.
   [junit4]   2> 744508 INFO  (qtp50170523-8841) [n:127.0.0.1:60969_solr 
c:nodes_used_collection s:shard2 r:core_node2 
x:nodes_used_collection_shard2_replica2] o.a.s.c.S.Request 
[nodes_used_collection_shard2_replica2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=0
   [junit4]   2> 744509 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.SyncStrategy Leader's attempt 
to sync with shard failed, moving to the next candidate
   [junit4]   2> 744509 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.ShardLeaderElectionContext We 
failed sync, but we have no versions - we can't sync in that case - we were 
active before, so become leader anyway
   [junit4]   2> 744513 INFO  (qtp854121126-8815) [n:127.0.0.1:40417_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica1] o.a.s.c.ShardLeaderElectionContext I 
am the new leader: 
https://127.0.0.1:40417/solr/nodes_used_collection_shard2_replica1/ shard2
   [junit4]   2> 744616 INFO  
(zkCallback-722-thread-1-processing-n:127.0.0.1:39538_solr) 
[n:127.0.0.1:39538_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 744616 INFO  
(zkCallback-710-thread-2-processing-n:127.0.0.1:47633_solr) 
[n:127.0.0.1:47633_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 744616 INFO  
(zkCallback-721-thread-2-processing-n:127.0.0.1:60969_solr) 
[n:127.0.0.1:60969_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 744616 INFO  
(zkCallback-723-thread-1-processing-n:127.0.0.1:40417_solr) 
[n:127.0.0.1:40417_solr    ] o.a.s.c.c.ZkSta

[...truncated too long message...]

ChaosMonkey monkey: stop shard! 54880
   [junit4]   2> 4707760 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[B6A88B4914AF140A])
 [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1629447914
   [junit4]   2> 4707767 INFO  (coreCloseExecutor-16385-thread-1) 
[n:127.0.0.1:54880_dxgi%2Fa c:target_collection s:shard2 r:core_node4 
x:target_collection_shard2_replica1] o.a.s.c.SolrCore 
[target_collection_shard2_replica1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@7e55d4f9
   [junit4]   2> 4707767 INFO  (coreCloseExecutor-16385-thread-1) 
[n:127.0.0.1:54880_dxgi%2Fa c:target_collection s:shard2 r:core_node4 
x:target_collection_shard2_replica1] o.a.s.h.CdcrRequestHandler Solr core is 
being closed - shutting down CDCR handler @ target_collection:shard2
   [junit4]   2> 4707780 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[B6A88B4914AF140A])
 [    ] o.a.s.c.Overseer Overseer 
(id=96932688936501281-127.0.0.1:54880_dxgi%2Fa-n_0000000007) closing
   [junit4]   2> 4707780 INFO  
(OverseerStateUpdate-96932688936501281-127.0.0.1:54880_dxgi%2Fa-n_0000000007) 
[n:127.0.0.1:54880_dxgi%2Fa    ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:54880_dxgi%2Fa
   [junit4]   2> 4709282 WARN  
(zkCallback-37223-thread-5-processing-n:127.0.0.1:54880_dxgi%2Fa) 
[n:127.0.0.1:54880_dxgi%2Fa    ] o.a.s.c.c.ZkStateReader ZooKeeper watch 
triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for 
/live_nodes]
   [junit4]   2> 4709283 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[B6A88B4914AF140A])
 [    ] o.e.j.s.ServerConnector Stopped ServerConnector@4a0f507d{SSL,[ssl, 
http/1.1]}{127.0.0.1:0}
   [junit4]   2> 4709283 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[B6A88B4914AF140A])
 [    ] o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@3b42fcec{/dxgi/a,null,UNAVAILABLE}
   [junit4]   2> 4709284 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[B6A88B4914AF140A])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testReplicationAfterRestart
   [junit4]   2> 4709285 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[B6A88B4914AF140A])
 [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53265 53265
   [junit4]   2> 4709467 INFO  (Thread-15974) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:53265 53265
   [junit4]   2> 4709468 WARN  (Thread-15974) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        35      /solr/aliases.json
   [junit4]   2>        18      /solr/configs/conf1
   [junit4]   2>        10      /solr/security.json
   [junit4]   2>        8       /solr/collections/tmp_collection/state.json
   [junit4]   2>        5       
/solr/collections/source_collection/leaders/shard1/leader
   [junit4]   2>        2       
/solr/collections/source_collection/leaders/shard2/leader
   [junit4]   2>        2       
/solr/collections/tmp_collection/leaders/shard2/leader
   [junit4]   2>        2       
/solr/collections/target_collection/leaders/shard2/leader
   [junit4]   2>        2       
/solr/collections/tmp_collection/leaders/shard1/leader
   [junit4]   2>        2       
/solr/collections/target_collection/leaders/shard1/leader
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        35      /solr/clusterprops.json
   [junit4]   2>        35      /solr/clusterstate.json
   [junit4]   2>        9       
/solr/collections/source_collection/cdcr/state/process
   [junit4]   2>        8       
/solr/collections/tmp_collection/cdcr/state/process
   [junit4]   2>        8       
/solr/collections/source_collection/cdcr/state/buffer
   [junit4]   2>        8       
/solr/collections/tmp_collection/cdcr/state/buffer
   [junit4]   2>        6       /solr/collections/source_collection/state.json
   [junit4]   2>        4       
/solr/collections/target_collection/cdcr/state/buffer
   [junit4]   2>        4       
/solr/collections/target_collection/cdcr/state/process
   [junit4]   2>        4       /solr/collections/target_collection/state.json
   [junit4]   2>        2       
/solr/collections/source_collection/leader_elect/shard1/election/96932688936501259-core_node4-n_0000000000
   [junit4]   2>        2       
/solr/overseer_elect/election/96932688936501281-127.0.0.1:54880_dxgi%2Fa-n_0000000007
   [junit4]   2>        2       
/solr/overseer_elect/election/96932688936501256-127.0.0.1:43664_dxgi%2Fa-n_0000000001
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        300     /solr/overseer/collection-queue-work
   [junit4]   2>        82      /solr/overseer/queue
   [junit4]   2>        35      /solr/collections
   [junit4]   2>        33      /solr/live_nodes
   [junit4]   2>        25      /solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> 4709469 INFO  
(SUITE-CdcrReplicationDistributedZkTest-seed#[B6A88B4914AF140A]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationDistributedZkTest_B6A88B4914AF140A-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, 
sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=no-NO, 
timezone=Pacific/Pohnpei
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 
1.8.0_102 (64-bit)/cpus=4,threads=1,free=160718320,total=524288000
   [junit4]   2> NOTE: All tests run in this JVM: 
[StatelessScriptUpdateProcessorFactoryTest, SpellPossibilityIteratorTest, 
TestPostingsSolrHighlighter, BlockJoinFacetSimpleTest, DistributedQueueTest, 
PreAnalyzedFieldTest, TestZkChroot, TestUseDocValuesAsStored2, 
TestAnalyzedSuggestions, TestSolrDeletionPolicy1, TestCloudRecovery, 
TestReRankQParserPlugin, TestSolr4Spatial2, DebugComponentTest, TestLFUCache, 
RequestLoggingTest, FieldMutatingUpdateProcessorTest, CoreAdminHandlerTest, 
ShowFileRequestHandlerTest, TestSolrConfigHandlerCloud, 
TlogReplayBufferedWhileIndexingTest, BadComponentTest, DOMUtilTest, 
CollectionsAPISolrJTest, TestHighFrequencyDictionaryFactory, DistanceUnitsTest, 
TestTolerantSearch, TimeZoneUtilsTest, LukeRequestHandlerTest, 
HttpPartitionTest, BigEndianAscendingWordDeserializerTest, PluginInfoTest, 
DistributedQueryComponentOptimizationTest, TestIndexingPerformance, 
TestCollapseQParserPlugin, TestCloudManagedSchema, 
DirectUpdateHandlerOptimizeTest, BlobRepositoryCloudTest, SolrTestCaseJ4Test, 
TestClusterStateMutator, MinimalSchemaTest, DistribCursorPagingTest, 
TestRecovery, LeaderElectionTest, TestLocalFSCloudBackupRestore, TestReload, 
ConnectionReuseTest, SOLR749Test, TestConfig, LeaderFailureAfterFreshStartTest, 
TestScoreJoinQPScore, CircularListTest, CdcrRequestHandlerTest, 
TestReplicaProperties, TestStressLiveNodes, TestExactSharedStatsCache, 
BadIndexSchemaTest, XmlUpdateRequestHandlerTest, 
OpenExchangeRatesOrgProviderTest, TestManagedSynonymFilterFactory, 
TestPushWriter, TestMacroExpander, AssignTest, WrapperMergePolicyFactoryTest, 
TestDFRSimilarityFactory, ExplicitHLLTest, TestWriterPerf, TestCryptoKeys, 
OverseerRolesTest, SecurityConfHandlerTest, TestSchemaNameResource, 
TestFieldSortValues, TestNumericTerms32, BasicAuthIntegrationTest, 
RollingRestartTest, ReplaceNodeTest, RecoveryZkTest, SuggesterWFSTTest, 
PrimUtilsTest, CdcrBootstrapTest, FieldAnalysisRequestHandlerTest, 
DataDrivenBlockJoinTest, HdfsDirectoryFactoryTest, DirectUpdateHandlerTest, 
TestSolrXml, PingRequestHandlerTest, NoCacheHeaderTest, SuggesterTSTTest, 
UpdateParamsTest, TestSolrCloudWithDelegationTokens, TestCollationField, 
TestNonDefinedSimilarityFactory, CleanupOldIndexTest, TestBinaryField, 
HdfsUnloadDistributedZkTest, HdfsChaosMonkeySafeLeaderTest, 
TestBlendedInfixSuggestions, CollectionStateFormat2Test, RequestHandlersTest, 
TestDocTermOrds, DeleteReplicaTest, TestIndexSearcher, DocValuesMultiTest, 
TermVectorComponentDistributedTest, MigrateRouteKeyTest, AutoCommitTest, 
TestDistributedGrouping, TestHashPartitioner, 
DistributedSpellCheckComponentTest, TestStressReorder, TestJoin, 
HardAutoCommitTest, TestCoreContainer, SimpleFacetsTest, TestSolr4Spatial, 
SolrCoreTest, StatsComponentTest, SpellCheckComponentTest, TestGroupingSearch, 
QueryElevationComponentTest, TestSort, TestFunctionQuery, TestLazyCores, 
TestBadConfig, CurrencyFieldOpenExchangeTest, 
DistributedQueryElevationComponentTest, CurrencyFieldXmlFileTest, 
SolrIndexSplitterTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, 
TestExtendedDismaxParser, SolrRequestParserTest, DocValuesTest, SuggesterTest, 
TestCSVLoader, PolyFieldTest, SolrCoreCheckLockOnStartupTest, TestUpdate, 
QueryEqualityTest, SortByFunctionTest, TestRemoteStreaming, SolrInfoMBeanTest, 
IndexBasedSpellCheckerTest, StandardRequestHandlerTest, 
DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, 
PrimitiveFieldTypeTest, TestOmitPositions, FileBasedSpellCheckerTest, 
TermsComponentTest, DocumentBuilderTest, TestValueSourceCache, 
TermVectorComponentTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, 
SolrPluginUtilsTest, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, 
JSONWriterTest, QueryParsingTest, UniqFieldsUpdateProcessorFactoryTest, 
BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, 
TestPHPSerializedResponseWriter, CSVRequestHandlerTest, 
TestQuerySenderListener, AlternateDirectoryTest, TestQuerySenderNoQuery, 
ResponseLogComponentTest, TestStressRecovery, TestMergePolicyConfig, 
TestSearchPerf, NumericFieldsTest, TestFuzzyAnalyzedSuggestions, 
TestSolrCoreProperties, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, 
TestCharFilters, SynonymTokenizerTest, TestXIncludeConfig, 
TestLMJelinekMercerSimilarityFactory, TestJmxMonitoredMap, 
URLClassifyProcessorTest, ClusterStateTest, TestDocumentBuilder, UUIDFieldTest, 
DistributedMLTComponentTest, TestHighlightDedupGrouping, 
TestSimpleTrackingShardHandler, AsyncCallRequestStatusResponseTest, 
CdcrReplicationDistributedZkTest]
   [junit4] Completed [527/651 (3!)] on J2 in 1774.99s, 11 tests, 1 failure <<< 
FAILURES!

[...truncated 61020 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to