Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/194/

4 tests failed.
FAILED:  
org.apache.solr.core.TestCoreContainer.testCoreInitFailuresFromEmptyContainer

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([CC32D335FB3C47BB]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestCoreContainer

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([CC32D335FB3C47BB]:0)


FAILED:  
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testSpecificConfigsets

Error Message:
KeeperErrorCode = NoNode for /collections/withconfigset2

Stack Trace:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode 
for /collections/withconfigset2
        at 
__randomizedtesting.SeedInfo.seed([CC32D335FB3C47BB:E14C9C6F0C18FDB7]:0)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:356)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:353)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testSpecificConfigsets(CollectionsAPIDistributedZkTest.java:425)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testCollectionsAPI

Error Message:
Expected to see collection awhollynewcollection_0 null Last available state: 
DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/4)={
   "replicationFactor":"2",   "shards":{     "shard1":{       
"range":"80000000-ffffffff",       "state":"active",       "replicas":{}},     
"shard2":{       "range":"0-7fffffff",       "state":"active",       
"replicas":{}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected to see collection awhollynewcollection_0
null
Last available state: 
DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/4)={
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false"}
        at 
__randomizedtesting.SeedInfo.seed([CC32D335FB3C47BB:8447A781FD0F682E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:236)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:496)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11967 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_CC32D335FB3C47BB-001/init-core-data-001
   [junit4]   2> 1312199 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1312200 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_CC32D335FB3C47BB-001/tempDir-001
   [junit4]   2> 1312200 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1312201 INFO  (Thread-3603) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1312201 INFO  (Thread-3603) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 1312301 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.a.s.c.ZkTestServer start zk server on port:34638
   [junit4]   2> 1312309 INFO  (jetty-launcher-876-thread-2) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1312309 INFO  (jetty-launcher-876-thread-1) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1312310 INFO  (jetty-launcher-876-thread-3) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1312310 INFO  (jetty-launcher-876-thread-4) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1312324 INFO  (jetty-launcher-876-thread-4) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@6b6c0534{/solr,null,AVAILABLE}
   [junit4]   2> 1312325 INFO  (jetty-launcher-876-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@3924e095{/solr,null,AVAILABLE}
   [junit4]   2> 1312327 INFO  (jetty-launcher-876-thread-4) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@65ffd482{SSL,[ssl, 
http/1.1]}{127.0.0.1:35620}
   [junit4]   2> 1312327 INFO  (jetty-launcher-876-thread-4) [    ] 
o.e.j.s.Server Started @1316358ms
   [junit4]   2> 1312327 INFO  (jetty-launcher-876-thread-1) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@1ca35a0b{SSL,[ssl, 
http/1.1]}{127.0.0.1:51690}
   [junit4]   2> 1312327 INFO  (jetty-launcher-876-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=35620}
   [junit4]   2> 1312327 INFO  (jetty-launcher-876-thread-1) [    ] 
o.e.j.s.Server Started @1316358ms
   [junit4]   2> 1312327 INFO  (jetty-launcher-876-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=51690}
   [junit4]   2> 1312327 ERROR (jetty-launcher-876-thread-4) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1312327 ERROR (jetty-launcher-876-thread-1) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1312327 INFO  (jetty-launcher-876-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 1312327 INFO  (jetty-launcher-876-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 1312327 INFO  (jetty-launcher-876-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1312327 INFO  (jetty-launcher-876-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1312327 INFO  (jetty-launcher-876-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1312327 INFO  (jetty-launcher-876-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1312328 INFO  (jetty-launcher-876-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-11-09T17:27:14.939Z
   [junit4]   2> 1312328 INFO  (jetty-launcher-876-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-11-09T17:27:14.939Z
   [junit4]   2> 1312338 INFO  (jetty-launcher-876-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@62d0c741{/solr,null,AVAILABLE}
   [junit4]   2> 1312340 INFO  (jetty-launcher-876-thread-2) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@6db3d17{SSL,[ssl, 
http/1.1]}{127.0.0.1:48116}
   [junit4]   2> 1312340 INFO  (jetty-launcher-876-thread-2) [    ] 
o.e.j.s.Server Started @1316371ms
   [junit4]   2> 1312340 INFO  (jetty-launcher-876-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=48116}
   [junit4]   2> 1312340 ERROR (jetty-launcher-876-thread-2) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1312340 INFO  (jetty-launcher-876-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 1312340 INFO  (jetty-launcher-876-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1312340 INFO  (jetty-launcher-876-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1312340 INFO  (jetty-launcher-876-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-11-09T17:27:14.951Z
   [junit4]   2> 1312342 INFO  (jetty-launcher-876-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@20186f0e{/solr,null,AVAILABLE}
   [junit4]   2> 1312344 INFO  (jetty-launcher-876-thread-3) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@6cab192{SSL,[ssl, 
http/1.1]}{127.0.0.1:32944}
   [junit4]   2> 1312344 INFO  (jetty-launcher-876-thread-3) [    ] 
o.e.j.s.Server Started @1316375ms
   [junit4]   2> 1312344 INFO  (jetty-launcher-876-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=32944}
   [junit4]   2> 1312344 ERROR (jetty-launcher-876-thread-3) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1312344 INFO  (jetty-launcher-876-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 1312344 INFO  (jetty-launcher-876-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1312344 INFO  (jetty-launcher-876-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1312344 INFO  (jetty-launcher-876-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-11-09T17:27:14.955Z
   [junit4]   2> 1312369 INFO  (jetty-launcher-876-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1312375 INFO  (jetty-launcher-876-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1312379 INFO  (jetty-launcher-876-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1312380 INFO  (jetty-launcher-876-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1312385 INFO  (jetty-launcher-876-thread-1) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1312386 INFO  (jetty-launcher-876-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34638/solr
   [junit4]   2> 1312387 INFO  (jetty-launcher-876-thread-4) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1312394 INFO  (jetty-launcher-876-thread-4) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34638/solr
   [junit4]   2> 1312396 INFO  (jetty-launcher-876-thread-2) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1312396 INFO  (jetty-launcher-876-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34638/solr
   [junit4]   2> 1312397 INFO  (jetty-launcher-876-thread-3) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1312407 INFO  (jetty-launcher-876-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34638/solr
   [junit4]   2> 1312423 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x1584a21207d0007, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1312434 INFO  (jetty-launcher-876-thread-1) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:51690_solr
   [junit4]   2> 1312435 INFO  (jetty-launcher-876-thread-1) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.Overseer Overseer 
(id=96908898127904776-127.0.0.1:51690_solr-n_0000000000) starting
   [junit4]   2> 1312454 INFO  (jetty-launcher-876-thread-3) 
[n:127.0.0.1:32944_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:32944_solr
   [junit4]   2> 1312459 INFO  
(zkCallback-898-thread-1-processing-n:127.0.0.1:35620_solr) 
[n:127.0.0.1:35620_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1312466 INFO  
(zkCallback-895-thread-1-processing-n:127.0.0.1:32944_solr) 
[n:127.0.0.1:32944_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1312466 INFO  
(zkCallback-897-thread-1-processing-n:127.0.0.1:48116_solr) 
[n:127.0.0.1:48116_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1312467 INFO  
(zkCallback-893-thread-1-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1312468 INFO  (jetty-launcher-876-thread-2) 
[n:127.0.0.1:48116_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:48116_solr
   [junit4]   2> 1312468 INFO  (jetty-launcher-876-thread-4) 
[n:127.0.0.1:35620_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:35620_solr
   [junit4]   2> 1312468 INFO  (jetty-launcher-876-thread-3) 
[n:127.0.0.1:32944_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_CC32D335FB3C47BB-001/tempDir-001/node3/.
   [junit4]   2> 1312470 INFO  (jetty-launcher-876-thread-1) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:51690_solr
   [junit4]   2> 1312470 INFO  
(zkCallback-895-thread-1-processing-n:127.0.0.1:32944_solr) 
[n:127.0.0.1:32944_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1312474 INFO  
(zkCallback-897-thread-1-processing-n:127.0.0.1:48116_solr) 
[n:127.0.0.1:48116_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1312477 INFO  
(zkCallback-898-thread-1-processing-n:127.0.0.1:35620_solr) 
[n:127.0.0.1:35620_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (3)
   [junit4]   2> 1312479 INFO  
(zkCallback-895-thread-1-processing-n:127.0.0.1:32944_solr) 
[n:127.0.0.1:32944_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 1312480 INFO  
(zkCallback-897-thread-1-processing-n:127.0.0.1:48116_solr) 
[n:127.0.0.1:48116_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 1312480 INFO  
(OverseerStateUpdate-96908898127904776-127.0.0.1:51690_solr-n_0000000000) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (3)
   [junit4]   2> 1312482 INFO  
(zkCallback-895-thread-1-processing-n:127.0.0.1:32944_solr) 
[n:127.0.0.1:32944_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 1312482 INFO  
(zkCallback-897-thread-1-processing-n:127.0.0.1:48116_solr) 
[n:127.0.0.1:48116_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 1312482 INFO  
(zkCallback-898-thread-1-processing-n:127.0.0.1:35620_solr) 
[n:127.0.0.1:35620_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 1312491 INFO  (jetty-launcher-876-thread-4) 
[n:127.0.0.1:35620_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_CC32D335FB3C47BB-001/tempDir-001/node4/.
   [junit4]   2> 1312493 INFO  (jetty-launcher-876-thread-1) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_CC32D335FB3C47BB-001/tempDir-001/node1/.
   [junit4]   2> 1312494 INFO  
(zkCallback-893-thread-2-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 1312495 INFO  (jetty-launcher-876-thread-2) 
[n:127.0.0.1:48116_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_CC32D335FB3C47BB-001/tempDir-001/node2/.
   [junit4]   2> 1312501 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1312541 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1312546 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1312549 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 1312561 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_46742_hdfs____.acthk3/webapp
   [junit4]   2> 1312972 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46742
   [junit4]   2> 1313097 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1313104 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 1313123 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_60641_datanode____r4xcio/webapp
   [junit4]   2> 1313560 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:60641
   [junit4]   2> 1313651 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1313654 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 1313673 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_48159_datanode____r3zsoo/webapp
   [junit4]   2> 1313837 INFO  (IPC Server handler 3 on 47963) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-d17c03e3-bdd0-42e9-aa7c-ac38003eacc5 node 
DatanodeRegistration(127.0.0.1:54270, 
datanodeUuid=44a353e9-9059-454a-96bc-eee8543815db, infoPort=35513, 
infoSecurePort=0, ipcPort=40672, 
storageInfo=lv=-56;cid=testClusterID;nsid=509662527;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 1313837 INFO  (IPC Server handler 3 on 47963) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-78b738d3-cd65-45ad-ac56-135081a4cc18 node 
DatanodeRegistration(127.0.0.1:54270, 
datanodeUuid=44a353e9-9059-454a-96bc-eee8543815db, infoPort=35513, 
infoSecurePort=0, ipcPort=40672, 
storageInfo=lv=-56;cid=testClusterID;nsid=509662527;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1314149 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CC32D335FB3C47BB]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:48159
   [junit4]   2> 1314302 INFO  (IPC Server handler 4 on 47963) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-224f6f47-1aa3-4ffb-946e-4359cb22dde5 node 
DatanodeRegistration(127.0.0.1:35172, 
datanodeUuid=7a4db8a7-a6f6-424b-a7c4-f2d26d695cfa, infoPort=58969, 
infoSecurePort=0, ipcPort=48636, 
storageInfo=lv=-56;cid=testClusterID;nsid=509662527;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 1314302 INFO  (IPC Server handler 4 on 47963) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-9cde1523-c0ec-4abd-9cfe-acd5e8a8c992 node 
DatanodeRegistration(127.0.0.1:35172, 
datanodeUuid=7a4db8a7-a6f6-424b-a7c4-f2d26d695cfa, infoPort=58969, 
infoSecurePort=0, ipcPort=48636, 
storageInfo=lv=-56;cid=testClusterID;nsid=509662527;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1314389 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[CC32D335FB3C47BB])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testTooManyReplicas
   [junit4]   2> 1314390 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[CC32D335FB3C47BB])
 [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1314455 INFO  (qtp1486891540-7896) [n:127.0.0.1:35620_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=10&collection.configName=conf&name=collection&action=CREATE&numShards=2&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1314459 INFO  
(OverseerThreadFactory-2871-thread-1-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
collection
   [junit4]   2> 1314459 WARN  
(OverseerThreadFactory-2871-thread-1-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.CreateCollectionCmd Specified 
replicationFactor of 10 on collection collection is higher than or equal to the 
number of Solr instances currently live or live and part of your 
createNodeSet(4). It's unusual to run two replica of the same slice on the same 
Solr-instance.
   [junit4]   2> 1314459 ERROR 
(OverseerThreadFactory-2871-thread-1-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.OverseerCollectionMessageHandler 
Collection: collection operation: create 
failed:org.apache.solr.common.SolrException: Cannot create collection 
collection. Value of maxShardsPerNode is 1, and the number of nodes currently 
live or live and part of your createNodeSet is 4. This allows a maximum of 4 to 
be created. Value of numShards is 2 and value of replicationFactor is 10. This 
requires 20 shards to be created (higher than the allowed number)
   [junit4]   2>        at 
org.apache.solr.cloud.CreateCollectionCmd.call(CreateCollectionCmd.java:148)
   [junit4]   2>        at 
org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:224)
   [junit4]   2>        at 
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:463)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1314464 INFO  (qtp1486891540-7896) [n:127.0.0.1:35620_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=10&collection.configName=conf&name=collection&action=CREATE&numShards=2&wt=javabin&version=2}
 status=400 QTime=8
   [junit4]   2> 1314466 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[CC32D335FB3C47BB])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testTooManyReplicas
   [junit4]   2> 1314550 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes-seed#[CC32D335FB3C47BB])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testCoresAreDistributedAcrossNodes
   [junit4]   2> 1314551 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes-seed#[CC32D335FB3C47BB])
 [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1314554 INFO  (qtp1486891540-7897) [n:127.0.0.1:35620_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&collection.configName=conf&name=nodes_used_collection&action=CREATE&numShards=2&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1314557 INFO  
(OverseerThreadFactory-2871-thread-2-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
nodes_used_collection
   [junit4]   2> 1314557 INFO  
(OverseerCollectionConfigSetProcessor-96908898127904776-127.0.0.1:51690_solr-n_0000000000)
 [n:127.0.0.1:51690_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 1314704 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1314731 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1314744 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1314746 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1314851 INFO  
(zkCallback-895-thread-1-processing-n:127.0.0.1:32944_solr) 
[n:127.0.0.1:32944_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1314851 INFO  
(zkCallback-897-thread-1-processing-n:127.0.0.1:48116_solr) 
[n:127.0.0.1:48116_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1314851 INFO  
(zkCallback-898-thread-1-processing-n:127.0.0.1:35620_solr) 
[n:127.0.0.1:35620_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1314851 INFO  
(zkCallback-893-thread-1-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1315744 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 1315747 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 1315754 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 1315788 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica1] 
o.a.s.s.IndexSchema [nodes_used_collection_shard2_replica1] Schema name=minimal
   [junit4]   2> 1315792 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.s.IndexSchema [nodes_used_collection_shard1_replica2] Schema name=minimal
   [junit4]   2> 1315795 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica1] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1315797 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1315797 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 1315798 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica1] 
o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard2_replica1' 
using configuration from collection nodes_used_collection
   [junit4]   2> 1315799 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard1_replica2' 
using configuration from collection nodes_used_collection
   [junit4]   2> 1315806 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.s.IndexSchema [nodes_used_collection_shard2_replica2] Schema name=minimal
   [junit4]   2> 1315809 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1315811 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:47963/data
   [junit4]   2> 1315811 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Solr 
Kerberos Authentication disabled
   [junit4]   2> 1315811 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.SolrCore 
[[nodes_used_collection_shard2_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_CC32D335FB3C47BB-001/tempDir-001/node1/nodes_used_collection_shard2_replica1],
 dataDir=[hdfs://localhost:47963/data/nodes_used_collection/core_node3/data/]
   [junit4]   2> 1315811 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:47963/data/nodes_used_collection/core_node3/data/snapshot_metadata
   [junit4]   2> 1315812 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica1] 
o.a.s.s.IndexSchema [nodes_used_collection_shard1_replica1] Schema name=minimal
   [junit4]   2> 1315814 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica1] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1315815 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard2_replica2' 
using configuration from collection nodes_used_collection
   [junit4]   2> 1315816 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:47963/data
   [junit4]   2> 1315816 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Solr 
Kerberos Authentication disabled
   [junit4]   2> 1315816 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.c.SolrCore 
[[nodes_used_collection_shard2_replica2] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_CC32D335FB3C47BB-001/tempDir-001/node3/nodes_used_collection_shard2_replica2],
 dataDir=[hdfs://localhost:47963/data/nodes_used_collection/core_node4/data/]
   [junit4]   2> 1315816 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:47963/data/nodes_used_collection/core_node4/data/snapshot_metadata
   [junit4]   2> 1315819 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica1] 
o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard1_replica1' 
using configuration from collection nodes_used_collection
   [junit4]   2> 1315819 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:47963/data
   [junit4]   2> 1315819 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Solr 
Kerberos Authentication disabled
   [junit4]   2> 1315819 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.c.SolrCore 
[[nodes_used_collection_shard1_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_CC32D335FB3C47BB-001/tempDir-001/node2/nodes_used_collection_shard1_replica1],
 dataDir=[hdfs://localhost:47963/data/nodes_used_collection/core_node2/data/]
   [junit4]   2> 1315819 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:47963/data/nodes_used_collection/core_node2/data/snapshot_metadata
   [junit4]   2> 1315820 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:47963/data
   [junit4]   2> 1315820 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Solr 
Kerberos Authentication disabled
   [junit4]   2> 1315820 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.SolrCore 
[[nodes_used_collection_shard1_replica2] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_CC32D335FB3C47BB-001/tempDir-001/node4/nodes_used_collection_shard1_replica2],
 dataDir=[hdfs://localhost:47963/data/nodes_used_collection/core_node1/data/]
   [junit4]   2> 1315820 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:47963/data/nodes_used_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 1315828 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1315828 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 1315835 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1315835 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 1315838 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1315838 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 1315839 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1315839 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 1316041 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 1316043 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 1316045 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 1316046 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 1316046 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:47963/data/nodes_used_collection/core_node4/data
   [junit4]   2> 1316047 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:47963/data/nodes_used_collection/core_node1/data
   [junit4]   2> 1316051 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:47963/data/nodes_used_collection/core_node2/data
   [junit4]   2> 1316054 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:47963/data/nodes_used_collection/core_node3/data
   [junit4]   2> 1316074 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:47963/data/nodes_used_collection/core_node4/data/index
   [junit4]   2> 1316078 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:47963/data/nodes_used_collection/core_node3/data/index
   [junit4]   2> 1316085 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:47963/data/nodes_used_collection/core_node1/data/index
   [junit4]   2> 1316088 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1316088 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 1316091 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1316092 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 1316095 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1316095 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 1316098 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 1316099 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 1316102 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:47963/data/nodes_used_collection/core_node2/data/index
   [junit4]   2> 1316107 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 1316112 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of 
slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1316112 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [16777216] will allocate [1] slabs and 
use ~[16777216] bytes
   [junit4]   2> 1316127 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache 
on write is disabled
   [junit4]   2> 1316245 INFO  (IPC Server handler 1 on 47963) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35172 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-d17c03e3-bdd0-42e9-aa7c-ac38003eacc5:NORMAL:127.0.0.1:54270|RBW],
 
ReplicaUC[[DISK]DS-224f6f47-1aa3-4ffb-946e-4359cb22dde5:NORMAL:127.0.0.1:35172|FINALIZED]]}
 size 0
   [junit4]   2> 1316247 INFO  (IPC Server handler 8 on 47963) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54270 is 
added to blk_1073741826_1002 size 71
   [junit4]   2> 1316250 INFO  (IPC Server handler 0 on 47963) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54270 is 
added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-9cde1523-c0ec-4abd-9cfe-acd5e8a8c992:NORMAL:127.0.0.1:35172|RBW],
 
ReplicaUC[[DISK]DS-78b738d3-cd65-45ad-ac56-135081a4cc18:NORMAL:127.0.0.1:54270|RBW]]}
 size 0
   [junit4]   2> 1316250 INFO  (IPC Server handler 5 on 47963) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35172 is 
added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-9cde1523-c0ec-4abd-9cfe-acd5e8a8c992:NORMAL:127.0.0.1:35172|RBW],
 
ReplicaUC[[DISK]DS-78b738d3-cd65-45ad-ac56-135081a4cc18:NORMAL:127.0.0.1:54270|RBW]]}
 size 0
   [junit4]   2> 1316250 INFO  (IPC Server handler 5 on 47963) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35172 is 
added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-9cde1523-c0ec-4abd-9cfe-acd5e8a8c992:NORMAL:127.0.0.1:35172|RBW],
 
ReplicaUC[[DISK]DS-78b738d3-cd65-45ad-ac56-135081a4cc18:NORMAL:127.0.0.1:54270|RBW]]}
 size 0
   [junit4]   2> 1316251 INFO  (IPC Server handler 6 on 47963) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54270 is 
added to blk_1073741825_1001 size 71
   [junit4]   2> 1316305 INFO  (IPC Server handler 0 on 47963) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35172 is 
added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-d17c03e3-bdd0-42e9-aa7c-ac38003eacc5:NORMAL:127.0.0.1:54270|RBW],
 
ReplicaUC[[DISK]DS-224f6f47-1aa3-4ffb-946e-4359cb22dde5:NORMAL:127.0.0.1:35172|RBW]]}
 size 0
   [junit4]   2> 1316305 INFO  (IPC Server handler 5 on 47963) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54270 is 
added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-d17c03e3-bdd0-42e9-aa7c-ac38003eacc5:NORMAL:127.0.0.1:54270|RBW],
 
ReplicaUC[[DISK]DS-224f6f47-1aa3-4ffb-946e-4359cb22dde5:NORMAL:127.0.0.1:35172|RBW]]}
 size 0
   [junit4]   2> 1316310 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1316310 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1316310 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.u.HdfsUpdateLog Initializing 
HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 1316314 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1316314 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1316314 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.u.HdfsUpdateLog Initializing 
HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 1316318 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1316318 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1316318 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.u.HdfsUpdateLog Initializing 
HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 1316332 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 1316332 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 1316343 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 1316343 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 1316351 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 1316351 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 1316353 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1316353 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1316353 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.u.HdfsUpdateLog Initializing 
HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 1316366 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@d03be0d[nodes_used_collection_shard1_replica2] main]
   [junit4]   2> 1316367 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1316367 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1316367 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.h.ReplicationHandler Commits 
will be reserved for  10000
   [junit4]   2> 1316371 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1550542374417989632
   [junit4]   2> 1316373 INFO  
(searcherExecutor-2879-thread-1-processing-n:127.0.0.1:35620_solr 
x:nodes_used_collection_shard1_replica2 s:shard1 c:nodes_used_collection 
r:core_node1) [n:127.0.0.1:35620_solr c:nodes_used_collection s:shard1 
r:core_node1 x:nodes_used_collection_shard1_replica2] o.a.s.c.SolrCore 
[nodes_used_collection_shard1_replica2] Registered new searcher 
Searcher@d03be0d[nodes_used_collection_shard1_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1316374 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@145aaca4[nodes_used_collection_shard2_replica1] main]
   [junit4]   2> 1316376 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1316376 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1316377 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.h.ReplicationHandler Commits 
will be reserved for  10000
   [junit4]   2> 1316377 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1550542374424281088
   [junit4]   2> 1316377 INFO  
(searcherExecutor-2878-thread-1-processing-n:127.0.0.1:51690_solr 
x:nodes_used_collection_shard2_replica1 s:shard2 c:nodes_used_collection 
r:core_node3) [n:127.0.0.1:51690_solr c:nodes_used_collection s:shard2 
r:core_node3 x:nodes_used_collection_shard2_replica1] o.a.s.c.SolrCore 
[nodes_used_collection_shard2_replica1] Registered new searcher 
Searcher@145aaca4[nodes_used_collection_shard2_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1316390 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 1316390 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 1316430 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.ShardLeaderElectionContext 
Waiting until we see more replicas up for shard shard1: total=2 found=1 
timeoutin=9999ms
   [junit4]   2> 1316431 INFO  
(zkCallback-898-thread-1-processing-n:127.0.0.1:35620_solr) 
[n:127.0.0.1:35620_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1316431 INFO  
(zkCallback-897-thread-1-processing-n:127.0.0.1:48116_solr) 
[n:127.0.0.1:48116_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1316431 INFO  
(zkCallback-893-thread-2-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1316431 INFO  
(zkCallback-895-thread-1-processing-n:127.0.0.1:32944_solr) 
[n:127.0.0.1:32944_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1316435 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.ShardLeaderElectionContext 
Waiting until we see more replicas up for shard shard2: total=2 found=1 
timeoutin=9999ms
   [junit4]   2> 1316436 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@4028db32[nodes_used_collection_shard2_replica2] main]
   [junit4]   2> 1316437 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1316437 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1316437 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.h.ReplicationHandler Commits 
will be reserved for  10000
   [junit4]   2> 1316438 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1550542374488244224
   [junit4]   2> 1316438 INFO  
(searcherExecutor-2880-thread-1-processing-n:127.0.0.1:32944_solr 
x:nodes_used_collection_shard2_replica2 s:shard2 c:nodes_used_collection 
r:core_node4) [n:127.0.0.1:32944_solr c:nodes_used_collection s:shard2 
r:core_node4 x:nodes_used_collection_shard2_replica2] o.a.s.c.SolrCore 
[nodes_used_collection_shard2_replica2] Registered new searcher 
Searcher@4028db32[nodes_used_collection_shard2_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1316447 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@90cf88[nodes_used_collection_shard1_replica1] main]
   [junit4]   2> 1316448 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1316449 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1316449 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.h.ReplicationHandler Commits 
will be reserved for  10000
   [junit4]   2> 1316449 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1550542374499778560
   [junit4]   2> 1316451 INFO  
(searcherExecutor-2881-thread-1-processing-n:127.0.0.1:48116_solr 
x:nodes_used_collection_shard1_replica1 s:shard1 c:nodes_used_collection 
r:core_node2) [n:127.0.0.1:48116_solr c:nodes_used_collection s:shard1 
r:core_node2 x:nodes_used_collection_shard1_replica1] o.a.s.c.SolrCore 
[nodes_used_collection_shard1_replica1] Registered new searcher 
Searcher@90cf88[nodes_used_collection_shard1_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1316538 INFO  
(zkCallback-895-thread-1-processing-n:127.0.0.1:32944_solr) 
[n:127.0.0.1:32944_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1316538 INFO  
(zkCallback-897-thread-1-processing-n:127.0.0.1:48116_solr) 
[n:127.0.0.1:48116_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1316539 INFO  
(zkCallback-893-thread-2-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1316539 INFO  
(zkCallback-898-thread-1-processing-n:127.0.0.1:35620_solr) 
[n:127.0.0.1:35620_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1316931 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.ShardLeaderElectionContext 
Enough replicas found to continue.
   [junit4]   2> 1316931 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.ShardLeaderElectionContext I 
may be the new leader - try and sync
   [junit4]   2> 1316931 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:35620/solr/nodes_used_collection_shard1_replica2/
   [junit4]   2> 1316931 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.u.PeerSync PeerSync: 
core=nodes_used_collection_shard1_replica2 url=https://127.0.0.1:35620/solr 
START 
replicas=[https://127.0.0.1:48116/solr/nodes_used_collection_shard1_replica1/] 
nUpdates=100
   [junit4]   2> 1316931 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.u.PeerSync PeerSync: 
core=nodes_used_collection_shard1_replica2 url=https://127.0.0.1:35620/solr 
DONE.  We have no versions.  sync failed.
   [junit4]   2> 1316936 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.ShardLeaderElectionContext 
Enough replicas found to continue.
   [junit4]   2> 1316936 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.ShardLeaderElectionContext I 
may be the new leader - try and sync
   [junit4]   2> 1316936 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:51690/solr/nodes_used_collection_shard2_replica1/
   [junit4]   2> 1316936 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.u.PeerSync PeerSync: 
core=nodes_used_collection_shard2_replica1 url=https://127.0.0.1:51690/solr 
START 
replicas=[https://127.0.0.1:32944/solr/nodes_used_collection_shard2_replica2/] 
nUpdates=100
   [junit4]   2> 1316936 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.u.PeerSync PeerSync: 
core=nodes_used_collection_shard2_replica1 url=https://127.0.0.1:51690/solr 
DONE.  We have no versions.  sync failed.
   [junit4]   2> 1316938 INFO  (qtp920440528-7901) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.c.S.Request 
[nodes_used_collection_shard1_replica1]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=0
   [junit4]   2> 1316939 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.SyncStrategy Leader's attempt 
to sync with shard failed, moving to the next candidate
   [junit4]   2> 1316939 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.ShardLeaderElectionContext We 
failed sync, but we have no versions - we can't sync in that case - we were 
active before, so become leader anyway
   [junit4]   2> 1316941 INFO  (qtp2011841402-7884) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.c.S.Request 
[nodes_used_collection_shard2_replica2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=0
   [junit4]   2> 1316942 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.SyncStrategy Leader's attempt 
to sync with shard failed, moving to the next candidate
   [junit4]   2> 1316942 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.ShardLeaderElectionContext We 
failed sync, but we have no versions - we can't sync in that case - we were 
active before, so become leader anyway
   [junit4]   2> 1316947 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.ShardLeaderElectionContext I 
am the new leader: 
https://127.0.0.1:51690/solr/nodes_used_collection_shard2_replica1/ shard2
   [junit4]   2> 1316948 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.ShardLeaderElectionContext I 
am the new leader: 
https://127.0.0.1:35620/solr/nodes_used_collection_shard1_replica2/ shard1
   [junit4]   2> 1317055 INFO  
(zkCallback-895-thread-1-processing-n:127.0.0.1:32944_solr) 
[n:127.0.0.1:32944_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1317055 INFO  
(zkCallback-897-thread-1-processing-n:127.0.0.1:48116_solr) 
[n:127.0.0.1:48116_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1317055 INFO  
(zkCallback-898-thread-1-processing-n:127.0.0.1:35620_solr) 
[n:127.0.0.1:35620_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1317058 INFO  
(zkCallback-893-thread-3-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1317099 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.c.ZkController I am the leader, 
no recovery necessary
   [junit4]   2> 1317099 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.c.ZkController I am the leader, 
no recovery necessary
   [junit4]   2> 1317101 INFO  (qtp1486891540-7894) [n:127.0.0.1:35620_solr 
c:nodes_used_collection s:shard1 r:core_node1 
x:nodes_used_collection_shard1_replica2] o.a.s.s.HttpSolrCall [admin] 
webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2}
 status=0 QTime=2398
   [junit4]   2> 1317103 INFO  (qtp146862523-7902) [n:127.0.0.1:51690_solr 
c:nodes_used_collection s:shard2 r:core_node3 
x:nodes_used_collection_shard2_replica1] o.a.s.s.HttpSolrCall [admin] 
webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2}
 status=0 QTime=2358
   [junit4]   2> 1317207 INFO  
(zkCallback-895-thread-1-processing-n:127.0.0.1:32944_solr) 
[n:127.0.0.1:32944_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1317207 INFO  
(zkCallback-897-thread-1-processing-n:127.0.0.1:48116_solr) 
[n:127.0.0.1:48116_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1317208 INFO  
(zkCallback-898-thread-1-processing-n:127.0.0.1:35620_solr) 
[n:127.0.0.1:35620_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1317208 INFO  
(zkCallback-893-thread-3-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1317449 INFO  (qtp2011841402-7916) [n:127.0.0.1:32944_solr 
c:nodes_used_collection s:shard2 r:core_node4 
x:nodes_used_collection_shard2_replica2] o.a.s.s.HttpSolrCall [admin] 
webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2}
 status=0 QTime=2703
   [junit4]   2> 1317455 INFO  (qtp920440528-7908) [n:127.0.0.1:48116_solr 
c:nodes_used_collection s:shard1 r:core_node2 
x:nodes_used_collection_shard1_replica1] o.a.s.s.HttpSolrCall [admin] 
webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2}
 status=0 QTime=2724
   [junit4]   2> 1317460 INFO  (qtp1486891540-7897) [n:127.0.0.1:35620_solr    
] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 1317560 INFO  
(zkCallback-895-thread-1-processing-n:127.0.0.1:32944_solr) 
[n:127.0.0.1:32944_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1317560 INFO  
(zkCallback-897-thread-1-processing-n:127.0.0.1:48116_solr) 
[n:127.0.0.1:48116_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1317560 INFO  
(zkCallback-893-thread-2-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1317560 INFO  
(zkCallback-898-thread-1-processing-n:127.0.0.1:35620_solr) 
[n:127.0.0.1:35620_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1318460 INFO  (qtp1486891540-7897) [n:127.0.0.1:35620_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=2&collection.configName=conf&name=nodes_used_collection&action=CREATE&numShards=2&wt=javabin&version=2}
 status=0 QTime=3905
   [junit4]   2> 1318461 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes-seed#[CC32D335FB3C47BB])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testCoresAreDistributedAcrossNodes
   [junit4]   2> 1318534 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[CC32D335FB3C47BB])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreationAndDeletion
   [junit4]   2> 1318536 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[CC32D335FB3C47BB])
 [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1318539 INFO  (qtp1486891540-7898) [n:127.0.0.1:35620_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params 
name=nodes_used_collection&action=DELETE&wt=javabin&version=2 and 
sendToOCPQueue=true
   [junit4]   2> 1318541 INFO  
(OverseerThreadFactory-2871-thread-3-processing-n:127.0.0.1:51690_solr) 
[n:127.0.0.1:51690_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing 
Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 1318547 INFO  
(OverseerCollectionConfigSetProcessor-96908898127904776-127.0.0.1:51690_solr-n_0000000000)
 [n:127.0.0.1:51690_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 1318547 INFO  (qtp1486891540-7896) [n:127.0.0.1:35620_solr    
] o.a.s.c.SolrCore [nodes_used_collection_shard1_replica2]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@1f1df52b
   [junit4]   2> 1318548 INFO  (qtp146862523-7903) [n:127.0.0.1:51690_solr    ] 
o.a.s.c.SolrCore [nodes_used_collection_shard2_replica1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@5691d403
   [junit4]   2> 1318550 INFO  (qtp920440528-7909) [n:127.0.0.1:48116_solr    ] 
o.a.s.c.SolrCore [nodes_used_collection_shard1_replica1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@ee45dd3
   [junit4]   2> 1318552 INFO  (qtp2011841402-7917) [n:127.0.0.1:32944_solr    
] o.a.s.c.SolrCore [nodes_used_collection_shard2_replica2]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@7678c71e
   [junit4]   2> 1318558 INFO  (qtp920440528-7909) [n:127.0.0.1:48116_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:47963/data/nodes_used_collection/core_node2/data/snapshot_metadata
   [junit4]   2> 1318559 INFO  (qtp920440528-7909) [n:127.0.0.1:48116_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:47963/data/nodes_used_collection/core_node2/data
   [junit4]   2> 1318561 INFO  (qtp920440528-7909) [n:127.0.0.1:48116_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:47963/data/nodes_used_collection/core_node2/data/index
   [junit4]   2> 1318568 INFO  (qtp146862523-7903) [n:127.0.0.1:51690_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:47963/data/nodes_used_collection/core_node3/data/snapshot_metadata
   [junit4]   2> 1318568 INFO  (qtp2011841402-7917) [n:127.0.0.1:32944_solr    
] o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:47963/data/nodes_used_collection/core_node4/data/index
   [junit4]   2> 1318569 INFO  (qtp146862523-7903) [n:127.0.0.1:51690_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:47963/data/nodes_used_collection/core_node3/data/index
   [junit4]   2> 1318569 INFO  (qtp146862523-7903) [n:127.0.0.1:51690_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:47963/data/nodes_used_collection/core_node3/data
   [junit4]   2> 1318574 INFO  (qtp1486891540-7896) [n:127.0.0.1:35620_solr    
] o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:47963/data/nodes_used_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 1318575 INFO  (qtp2011841402-7917) [n:127.0.0.1:32944_solr    
] o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:47963/data/nodes_used_collection/core_node4/data/snapshot_metadata
   [junit4]   2> 1318575 INFO  (qtp2011841402-7917) [n:127.0.0.1:32944_solr    
] o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:47963/data/nodes_used_collection/core_node4/data
   [junit4]   2> 1318579 INFO  (qtp1486891540-7896) [n:127.0.0.1:35620_solr    
] o.a.s.s.h.HdfsDirectory Closing h

[...truncated too long message...]

  at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>         at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    2) Thread[id=40629, name=solr-idle-connections-evictor, 
state=TIMED_WAITING, group=TGRP-TestCoreContainer]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at 
org.apache.solr.update.UpdateShardHandler$IdleConnectionsEvictor$1.run(UpdateShardHandler.java:187)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    3) Thread[id=40630, name=Thread-35572, state=WAITING, 
group=TGRP-TestCoreContainer]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Object.wait(Object.java:502)
   [junit4]   2>         at 
org.apache.solr.core.CloserThread.run(CoreContainer.java:1274)
   [junit4]   2>    4) Thread[id=40628, name=solr-idle-connections-evictor, 
state=TIMED_WAITING, group=TGRP-TestCoreContainer]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at 
org.apache.solr.update.UpdateShardHandler$IdleConnectionsEvictor$1.run(UpdateShardHandler.java:187)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> ??? 09, 2016 4:39:35 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=40627, 
name=TEST-TestCoreContainer.testCoreInitFailuresFromEmptyContainer-seed#[CC32D335FB3C47BB],
 state=WAITING, group=TGRP-TestCoreContainer]
   [junit4]   2>    2) Thread[id=40629, name=solr-idle-connections-evictor, 
state=TIMED_WAITING, group=TGRP-TestCoreContainer]
   [junit4]   2>    3) Thread[id=40630, name=Thread-35572, state=WAITING, 
group=TGRP-TestCoreContainer]
   [junit4]   2>    4) Thread[id=40628, name=solr-idle-connections-evictor, 
state=TIMED_WAITING, group=TGRP-TestCoreContainer]
   [junit4]   2> 9252985 INFO  
(TEST-TestCoreContainer.testCoreInitFailuresFromEmptyContainer-seed#[CC32D335FB3C47BB])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testCoreInitFailuresFromEmptyContainer
   [junit4]   2> 9252986 INFO  
(SUITE-TestCoreContainer-seed#[CC32D335FB3C47BB]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Nov 09, 2016 7:39:35 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: 
codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
 chunkSize=3, maxDocsPerChunk=10, blockSize=648), 
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
 chunkSize=3, blockSize=648)), sim=ClassicSimilarity, locale=sr, 
timezone=America/Fortaleza
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 
1.8.0_102 (64-bit)/cpus=4,threads=1,free=295178848,total=529006592
   [junit4]   2> NOTE: All tests run in this JVM: [TestSmileRequest, 
TestSystemIdResolver, TestSolrIndexConfig, TestBackupRepositoryFactory, 
RestartWhileUpdatingTest, TestRequestStatusCollectionAPI, 
TestDocBasedVersionConstraints, TestJsonFacets, TestCloudPseudoReturnFields, 
TestSQLHandlerNonCloud, ChaosMonkeySafeLeaderTest, 
TestManagedStopFilterFactory, TestAuthorizationFramework, TestTrie, 
TestPartialUpdateDeduplication, TestLuceneMatchVersion, SimplePostToolTest, 
BlockDirectoryTest, DeleteNodeTest, TestCollectionAPI, 
TestHdfsCloudBackupRestore, AnalyticsQueryTest, DistributedMLTComponentTest, 
TestScoreJoinQPNoScore, DistributedTermsComponentTest, SaslZkACLProviderTest, 
TestGraphTermsQParserPlugin, LoggingHandlerTest, 
TestOverriddenPrefixQueryForCustomFieldType, OpenCloseCoreStressTest, 
SoftAutoCommitTest, SearchHandlerTest, 
LeaderInitiatedRecoveryOnShardRestartTest, DocumentBuilderTest, 
TestSimpleTrackingShardHandler, RAMDirectoryFactoryTest, 
SharedFSAutoReplicaFailoverUtilsTest, CloudMLTQParserTest, 
UUIDUpdateProcessorFallbackTest, HdfsNNFailoverTest, TestBulkSchemaAPI, 
TestSlowCompositeReaderWrapper, TestQuerySenderListener, 
SpellingQueryConverterTest, SliceStateTest, 
TestPerFieldSimilarityWithDefaultOverride, TestStandardQParsers, 
TestDistribIDF, DocValuesMissingTest, RulesTest, TestCustomDocTransformer, 
FileUtilsTest, TestRebalanceLeaders, TestSweetSpotSimilarityFactory, 
HdfsTlogReplayBufferedWhileIndexingTest, IndexSchemaRuntimeFieldTest, 
JsonLoaderTest, DeleteStatusTest, TestSortByMinMaxFunction, TestQueryUtils, 
TestFieldCacheReopen, RemoteQueryErrorTest, ExitableDirectoryReaderTest, 
TestLMDirichletSimilarityFactory, CoreAdminCreateDiscoverTest, 
BasicFunctionalityTest, TestSolrDynamicMBean, TestDFISimilarityFactory, 
TestAnalyzeInfixSuggestions, DistributedIntervalFacetingTest, 
TestSurroundQueryParser, HdfsRestartWhileUpdatingTest, 
TestFieldCacheSortRandom, TestCoreContainer]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCoreContainer 
-Dtests.seed=CC32D335FB3C47BB -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=sr -Dtests.timezone=America/Fortaleza -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | TestCoreContainer (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 
7200000 msec).
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([CC32D335FB3C47BB]:0)
   [junit4] Completed [618/651 (2!)] on J1 in 7220.25s, 1 test, 2 errors <<< 
FAILURES!

[...truncated 60755 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to