[JENKINS] Lucene-Solr-NightlyTests-6.4 - Build # 4 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.4/4/

6 tests failed.
FAILED:  
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testReplicationStartStop

Error Message:
Timeout while trying to assert number of documents @ target_collection

Stack Trace:
java.lang.AssertionError: Timeout while trying to assert number of documents @ 
target_collection
        at 
__randomizedtesting.SeedInfo.seed([A7B429E85CFE5361:2477F77B4826D8E8]:0)
        at 
org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:271)
        at 
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testReplicationStartStop(CdcrReplicationDistributedZkTest.java:173)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: expected:<10> but was:<4>
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at 
org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:262)
        ... 42 more


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
        at 
__randomizedtesting.SeedInfo.seed([A7B429E85CFE5361:2FE01632F2023E99]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:309)
        at 
org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:127)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
  at 
org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:130)  
at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:74) 
 at org.apache.solr.update.HdfsUpdateLog.ensureLog(HdfsUpdateLog.java:313)  at 
org.apache.solr.update.UpdateLog.deleteByQuery(UpdateLog.java:543)  at 
org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:464)
  at 
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:78)
  at 
org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:52)
  at 
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:960)
  at 
org.apache.solr.update.processor.DistributedUpdateProcessor.versionDeleteByQuery(DistributedUpdateProcessor.java:1455)
  at 
org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:1348)
  at 
org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:1157)
  at 
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:124)
  at 
org.apache.solr.handler.loader.JavabinLoader.delete(JavabinLoader.java:159)  at 
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:112)
  at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:54)  
at 
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
  at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
  at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)
  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2306)  at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)  at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
  at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:110)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) 
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)  
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) 
 at org.eclipse.jetty.server.Server.handle(Server.java:534)  at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) 
 at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
  at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
  at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
  at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
  at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) 
 at java.lang.Thread.run(Thread.java:745)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not 
released!!! [HdfsTransactionLog]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
        at 
org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:130)
        at 
org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:74)
        at 
org.apache.solr.update.HdfsUpdateLog.ensureLog(HdfsUpdateLog.java:313)
        at org.apache.solr.update.UpdateLog.deleteByQuery(UpdateLog.java:543)
        at 
org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:464)
        at 
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:78)
        at 
org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:52)
        at 
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:960)
        at 
org.apache.solr.update.processor.DistributedUpdateProcessor.versionDeleteByQuery(DistributedUpdateProcessor.java:1455)
        at 
org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:1348)
        at 
org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:1157)
        at 
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:124)
        at 
org.apache.solr.handler.loader.JavabinLoader.delete(JavabinLoader.java:159)
        at 
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:112)
        at 
org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:54)
        at 
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
        at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2306)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:110)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:745)


        at __randomizedtesting.SeedInfo.seed([A7B429E85CFE5361]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:269)
        at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testCollectionsAPI

Error Message:
Expected to see collection awhollynewcollection_0 null Last available state: 
DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/2)={
   "replicationFactor":"1",   "shards":{     "shard1":{       
"range":"80000000-bfffffff",       "state":"active",       "replicas":{}},     
"shard2":{       "range":"c0000000-ffffffff",       "state":"active",       
"replicas":{}},     "shard3":{       "range":"0-3fffffff",       
"state":"active",       "replicas":{}},     "shard4":{       
"range":"40000000-7fffffff",       "state":"active",       "replicas":{}}},   
"router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected to see collection awhollynewcollection_0
null
Last available state: 
DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/2)={
  "replicationFactor":"1",
  "shards":{
    "shard1":{
      "range":"80000000-bfffffff",
      "state":"active",
      "replicas":{}},
    "shard2":{
      "range":"c0000000-ffffffff",
      "state":"active",
      "replicas":{}},
    "shard3":{
      "range":"0-3fffffff",
      "state":"active",
      "replicas":{}},
    "shard4":{
      "range":"40000000-7fffffff",
      "state":"active",
      "replicas":{}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false"}
        at 
__randomizedtesting.SeedInfo.seed([A7B429E85CFE5361:EFC15D5C5ACD7CF4]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:265)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:496)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testSpecificConfigsets

Error Message:
KeeperErrorCode = NoNode for /collections/withconfigset2

Stack Trace:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode 
for /collections/withconfigset2
        at 
__randomizedtesting.SeedInfo.seed([A7B429E85CFE5361:8ACA66B2ABDAE96D]:0)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:356)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:353)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testSpecificConfigsets(CollectionsAPIDistributedZkTest.java:425)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
__randomizedtesting.SeedInfo.seed([A7B429E85CFE5361:2FE01632F2023E99]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at 
org.apache.solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest.test(HdfsWriteToMultipleCollectionsTest.java:137)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11381 lines...]
   [junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/init-core-data-001
   [junit4]   2> 252827 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 252827 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 255036 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your 
platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 256658 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 257681 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via 
org.mortbay.log.Slf4jLog
   [junit4]   2> 257893 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 258196 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 258632 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_43944_hdfs____62wp50/webapp
   [junit4]   2> 260807 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43944
   [junit4]   2> 263793 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 263798 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 263813 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_47113_datanode____r2xms7/webapp
   [junit4]   2> 264611 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47113
   [junit4]   2> 265984 WARN  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 265986 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 265996 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_55727_datanode____.ox5f5p/webapp
   [junit4]   2> 267190 INFO  
(SUITE-SharedFSAutoReplicaFailoverTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:55727
   [junit4]   2> 269623 INFO  (IPC Server handler 4 on 50553) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-72c7baad-9adb-4328-a2b3-d5583b05db08 node 
DatanodeRegistration(127.0.0.1:34847, 
datanodeUuid=e33c1b9d-f0b4-4132-958a-370f4671ac23, infoPort=45811, 
infoSecurePort=0, ipcPort=39999, 
storageInfo=lv=-56;cid=testClusterID;nsid=1805680937;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 184 msecs
   [junit4]   2> 269624 INFO  (IPC Server handler 9 on 50553) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-87fa2927-12f1-46c0-acff-b7fb8230f689 node 
DatanodeRegistration(127.0.0.1:53510, 
datanodeUuid=21f08e4a-c712-4bc1-ad53-b56408ec88a7, infoPort=47093, 
infoSecurePort=0, ipcPort=48679, 
storageInfo=lv=-56;cid=testClusterID;nsid=1805680937;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 269624 INFO  (IPC Server handler 4 on 50553) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-eccbb487-9d61-4016-bd93-5f40e890fda5 node 
DatanodeRegistration(127.0.0.1:34847, 
datanodeUuid=e33c1b9d-f0b4-4132-958a-370f4671ac23, infoPort=45811, 
infoSecurePort=0, ipcPort=39999, 
storageInfo=lv=-56;cid=testClusterID;nsid=1805680937;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 269624 INFO  (IPC Server handler 9 on 50553) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-0db8a561-9f92-4569-8546-05f32126e9e5 node 
DatanodeRegistration(127.0.0.1:53510, 
datanodeUuid=21f08e4a-c712-4bc1-ad53-b56408ec88a7, infoPort=47093, 
infoSecurePort=0, ipcPort=48679, 
storageInfo=lv=-56;cid=testClusterID;nsid=1805680937;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 270240 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 270243 INFO  (Thread-549) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 270243 INFO  (Thread-549) [    ] o.a.s.c.ZkTestServer Starting 
server
   [junit4]   2> 270343 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:56719
   [junit4]   2> 270514 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 270516 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 270517 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 270518 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 270522 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 270523 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 270524 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 270524 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 270525 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 270526 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 270527 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 270703 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/control-001/cores/collection1
   [junit4]   2> 270705 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 270780 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@7889fdf8{/,null,AVAILABLE}
   [junit4]   2> 270780 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@5b77c8ca{HTTP/1.1,[http/1.1]}{127.0.0.1:57313}
   [junit4]   2> 270780 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.e.j.s.Server Started @274346ms
   [junit4]   2> 270780 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=57313, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/control-001/cores}
   [junit4]   2> 270781 ERROR 
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 270781 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.4.0
   [junit4]   2> 270781 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 270781 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 270781 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-01-18T20:44:16.482Z
   [junit4]   2> 270862 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 270862 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/control-001/solr.xml
   [junit4]   2> 270915 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 270926 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56719/solr
   [junit4]   2> 271243 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:57313_    ] o.a.s.c.OverseerElectionContext I am going to be the 
leader 127.0.0.1:57313_
   [junit4]   2> 271244 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:57313_    ] o.a.s.c.Overseer Overseer 
(id=97306034565873668-127.0.0.1:57313_-n_0000000000) starting
   [junit4]   2> 271282 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:57313_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:57313_
   [junit4]   2> 271285 INFO  
(zkCallback-860-thread-1-processing-n:127.0.0.1:57313_) [n:127.0.0.1:57313_    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 271367 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:57313_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/control-001/cores
   [junit4]   2> 271367 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:57313_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 271400 INFO  
(OverseerStateUpdate-97306034565873668-127.0.0.1:57313_-n_0000000000) 
[n:127.0.0.1:57313_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard 
shard=shard1
   [junit4]   2> 271450 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 271527 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 271838 WARN  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
[collection1] default search field in schema is text. WARNING: Deprecated, 
please use 'df' on request instead.
   [junit4]   2> 271839 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 271944 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1' using configuration from collection 
control_collection
   [junit4]   2> 272304 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:50553/solr_hdfs_home
   [junit4]   2> 272304 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 272304 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] o.a.s.c.SolrCore 
[[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/control-001/cores/collection1],
 
dataDir=[hdfs://localhost:50553/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 272305 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ad3006c
   [junit4]   2> 272305 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:50553/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 272350 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 272350 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 272350 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 276353 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 276366 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:50553/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 276456 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:50553/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 276463 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 276463 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 276463 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 276471 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 276471 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: 
minMergeSize=1677721, mergeFactor=2, maxMergeSize=2147483648, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=1.0]
   [junit4]   2> 277142 INFO  (IPC Server handler 4 on 50553) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:34847 is 
added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-72c7baad-9adb-4328-a2b3-d5583b05db08:NORMAL:127.0.0.1:34847|RBW],
 
ReplicaUC[[DISK]DS-87fa2927-12f1-46c0-acff-b7fb8230f689:NORMAL:127.0.0.1:53510|RBW]]}
 size 71
   [junit4]   2> 277144 INFO  (IPC Server handler 9 on 50553) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53510 is 
added to blk_1073741825_1001 size 71
   [junit4]   2> 277718 WARN  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 277786 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 277786 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 277786 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] o.a.s.u.HdfsUpdateLog 
Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 277810 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 277810 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 277821 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=5, maxMergeAtOnceExplicit=8, maxMergedSegmentMB=2.1787109375, 
floorSegmentMB=1.013671875, forceMergeDeletesPctAllowed=8.500118411867492, 
segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.4695422095545214
   [junit4]   2> 278114 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@7c776ca5[collection1] main]
   [junit4]   2> 278115 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 278115 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 278118 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 278121 INFO  
(searcherExecutor-624-thread-1-processing-n:127.0.0.1:57313_ x:collection1 
c:control_collection) [n:127.0.0.1:57313_ c:control_collection   x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@7c776ca5[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 278133 INFO  
(coreLoadExecutor-623-thread-1-processing-n:127.0.0.1:57313_) 
[n:127.0.0.1:57313_ c:control_collection   x:collection1] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1556896561322000384
   [junit4]   2> 278156 INFO  
(coreZkRegister-616-thread-1-processing-n:127.0.0.1:57313_ x:collection1 
c:control_collection) [n:127.0.0.1:57313_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas 
found to continue.
   [junit4]   2> 278156 INFO  
(coreZkRegister-616-thread-1-processing-n:127.0.0.1:57313_ x:collection1 
c:control_collection) [n:127.0.0.1:57313_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new 
leader - try and sync
   [junit4]   2> 278156 INFO  
(coreZkRegister-616-thread-1-processing-n:127.0.0.1:57313_ x:collection1 
c:control_collection) [n:127.0.0.1:57313_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:57313/collection1/
   [junit4]   2> 278156 INFO  
(coreZkRegister-616-thread-1-processing-n:127.0.0.1:57313_ x:collection1 
c:control_collection) [n:127.0.0.1:57313_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync 
replicas to me
   [junit4]   2> 278156 INFO  
(coreZkRegister-616-thread-1-processing-n:127.0.0.1:57313_ x:collection1 
c:control_collection) [n:127.0.0.1:57313_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
http://127.0.0.1:57313/collection1/ has no replicas
   [junit4]   2> 278160 INFO  
(coreZkRegister-616-thread-1-processing-n:127.0.0.1:57313_ x:collection1 
c:control_collection) [n:127.0.0.1:57313_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new 
leader: http://127.0.0.1:57313/collection1/ shard1
   [junit4]   2> 278311 INFO  
(coreZkRegister-616-thread-1-processing-n:127.0.0.1:57313_ x:collection1 
c:control_collection) [n:127.0.0.1:57313_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery 
necessary
   [junit4]   2> 278449 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 278452 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:56719/solr ready
   [junit4]   2> 278452 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection 
loss:false
   [junit4]   2> 278465 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 278670 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/shard-1-001/cores/collection1
   [junit4]   2> 278670 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/shard-1-001
   [junit4]   2> 278683 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 278684 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@1ecbafe5{/,null,AVAILABLE}
   [junit4]   2> 278684 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@2d4a1371{HTTP/1.1,[http/1.1]}{127.0.0.1:57619}
   [junit4]   2> 278684 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.e.j.s.Server Started @282251ms
   [junit4]   2> 278685 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {solrconfig=solrconfig.xml, 
hostContext=/, hostPort=57619, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/shard-1-001/cores}
   [junit4]   2> 278702 ERROR 
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 278726 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.4.0
   [junit4]   2> 278726 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 278726 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 278726 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-01-18T20:44:24.427Z
   [junit4]   2> 278792 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 278792 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/shard-1-001/solr.xml
   [junit4]   2> 278798 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 278799 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56719/solr
   [junit4]   2> 278805 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:57619_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 278807 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:57619_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:57619_
   [junit4]   2> 278819 INFO  (zkCallback-864-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 278819 INFO  
(zkCallback-860-thread-1-processing-n:127.0.0.1:57313_) [n:127.0.0.1:57313_    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 278825 INFO  
(zkCallback-870-thread-1-processing-n:127.0.0.1:57619_) [n:127.0.0.1:57619_    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 278888 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:57619_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/shard-1-001/cores
   [junit4]   2> 278889 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:57619_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 278895 INFO  
(OverseerStateUpdate-97306034565873668-127.0.0.1:57313_-n_0000000000) 
[n:127.0.0.1:57313_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard 
shard=shard2
   [junit4]   2> 279100 INFO  
(zkCallback-870-thread-1-processing-n:127.0.0.1:57619_) [n:127.0.0.1:57619_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 279990 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using 
Lucene MatchVersion: 6.4.0
   [junit4]   2> 280101 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 280177 WARN  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] default search field in schema is text. WARNING: Deprecated, 
please use 'df' on request instead.
   [junit4]   2> 280180 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded 
schema test/1.6 with uniqueid field id
   [junit4]   2> 280260 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 280260 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:50553/solr_hdfs_home
   [junit4]   2> 280271 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
Solr Kerberos Authentication disabled
   [junit4]   2> 280271 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.SolrCore 
[[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/shard-1-001/cores/collection1],
 dataDir=[hdfs://localhost:50553/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 280272 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX 
monitoring is enabled. Adding Solr mbeans to JMX Server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@5ad3006c
   [junit4]   2> 280272 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
creating directory factory for path 
hdfs://localhost:50553/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 280358 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 280358 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
Block cache target memory usage, slab size of [8388608] will allocate [1] slabs 
and use ~[8388608] bytes
   [junit4]   2> 280358 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
Creating new single instance HDFS BlockCache
   [junit4]   2> 280375 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory 
Block cache on write is disabled
   [junit4]   2> 280393 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
creating directory factory for path 
hdfs://localhost:50553/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 280678 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
creating directory factory for path 
hdfs://localhost:50553/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 280726 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 280726 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
Block cache target memory usage, slab size of [8388608] will allocate [1] slabs 
and use ~[8388608] bytes
   [junit4]   2> 280726 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
Creating new single instance HDFS BlockCache
   [junit4]   2> 280738 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory 
Block cache on write is disabled
   [junit4]   2> 280738 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy 
RandomMergePolicy wrapping class 
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: 
minMergeSize=1677721, mergeFactor=2, maxMergeSize=2147483648, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=1.0]
   [junit4]   2> 280895 INFO  (IPC Server handler 9 on 50553) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:34847 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-87fa2927-12f1-46c0-acff-b7fb8230f689:NORMAL:127.0.0.1:53510|RBW],
 
ReplicaUC[[DISK]DS-eccbb487-9d61-4016-bd93-5f40e890fda5:NORMAL:127.0.0.1:34847|FINALIZED]]}
 size 0
   [junit4]   2> 280902 INFO  (IPC Server handler 1 on 50553) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53510 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-eccbb487-9d61-4016-bd93-5f40e890fda5:NORMAL:127.0.0.1:34847|FINALIZED],
 
ReplicaUC[[DISK]DS-0db8a561-9f92-4569-8546-05f32126e9e5:NORMAL:127.0.0.1:53510|FINALIZED]]}
 size 0
   [junit4]   2> 280949 WARN  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.c.RequestHandlers 
INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class 
= DumpRequestHandler,attributes = {initParams=a, name=/dump, 
class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 281136 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using 
UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 281136 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 281136 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog 
Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 281145 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard 
AutoCommit: disabled
   [junit4]   2> 281146 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft 
AutoCommit: disabled
   [junit4]   2> 281161 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy 
RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: 
[TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=8, 
maxMergedSegmentMB=2.1787109375, floorSegmentMB=1.013671875, 
forceMergeDeletesPctAllowed=8.500118411867492, segmentsPerTier=46.0, 
maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4695422095545214
   [junit4]   2> 281264 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher 
Opening [Searcher@95450a2[collection1] main]
   [junit4]   2> 281265 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 281265 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 281266 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler 
Commits will be reserved for  10000
   [junit4]   2> 281267 INFO  
(searcherExecutor-635-thread-1-processing-n:127.0.0.1:57619_ x:collection1 
c:collection1) [n:127.0.0.1:57619_ c:collection1   x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@95450a2[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 281279 INFO  
(coreLoadExecutor-634-thread-1-processing-n:127.0.0.1:57619_) 
[n:127.0.0.1:57619_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not 
find max version in index or recent updates, using new clock 1556896564620820480
   [junit4]   2> 281312 INFO  
(coreZkRegister-629-thread-1-processing-n:127.0.0.1:57619_ x:collection1 
c:collection1) [n:127.0.0.1:57619_ c:collection1 s:shard2 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to 
continue.
   [junit4]   2> 281312 INFO  
(coreZkRegister-629-thread-1-processing-n:127.0.0.1:57619_ x:collection1 
c:collection1) [n:127.0.0.1:57619_ c:collection1 s:shard2 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try 
and sync
   [junit4]   2> 281312 INFO  
(coreZkRegister-629-thread-1-processing-n:127.0.0.1:57619_ x:collection1 
c:collection1) [n:127.0.0.1:57619_ c:collection1 s:shard2 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:57619/collection1/
   [junit4]   2> 281312 INFO  
(coreZkRegister-629-thread-1-processing-n:127.0.0.1:57619_ x:collection1 
c:collection1) [n:127.0.0.1:57619_ c:collection1 s:shard2 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 281312 INFO  
(coreZkRegister-629-thread-1-processing-n:127.0.0.1:57619_ x:collection1 
c:collection1) [n:127.0.0.1:57619_ c:collection1 s:shard2 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:57619/collection1/ has no 
replicas
   [junit4]   2> 281313 INFO  
(zkCallback-870-thread-1-processing-n:127.0.0.1:57619_) [n:127.0.0.1:57619_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 281316 INFO  
(coreZkRegister-629-thread-1-processing-n:127.0.0.1:57619_ x:collection1 
c:collection1) [n:127.0.0.1:57619_ c:collection1 s:shard2 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:57619/collection1/ shard2
   [junit4]   2> 281418 INFO  
(zkCallback-870-thread-1-processing-n:127.0.0.1:57619_) [n:127.0.0.1:57619_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 281467 INFO  
(coreZkRegister-629-thread-1-processing-n:127.0.0.1:57619_ x:collection1 
c:collection1) [n:127.0.0.1:57619_ c:collection1 s:shard2 r:core_node1 
x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 281615 INFO  
(zkCallback-870-thread-1-processing-n:127.0.0.1:57619_) [n:127.0.0.1:57619_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 281765 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/shard-2-001/cores/collection1
   [junit4]   2> 281766 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/shard-2-001
   [junit4]   2> 281768 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 281768 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@54a86d3c{/,null,AVAILABLE}
   [junit4]   2> 281769 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@723b0377{HTTP/1.1,[http/1.1]}{127.0.0.1:38221}
   [junit4]   2> 281769 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.e.j.s.Server Started @285335ms
   [junit4]   2> 281769 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {solrconfig=solrconfig.xml, 
hostContext=/, hostPort=38221, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/shard-2-001/cores}
   [junit4]   2> 281769 ERROR 
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 281776 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.4.0
   [junit4]   2> 281776 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 281776 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 281776 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-01-18T20:44:27.477Z
   [junit4]   2> 281786 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 281786 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/shard-2-001/solr.xml
   [junit4]   2> 281821 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 281832 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56719/solr
   [junit4]   2> 281884 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:38221_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 281888 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:38221_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:38221_
   [junit4]   2> 281901 INFO  
(zkCallback-860-thread-3-processing-n:127.0.0.1:57313_) [n:127.0.0.1:57313_    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 281905 INFO  
(zkCallback-876-thread-1-processing-n:127.0.0.1:38221_) [n:127.0.0.1:38221_    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 281901 INFO  (zkCallback-864-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 281901 INFO  
(zkCallback-870-thread-1-processing-n:127.0.0.1:57619_) [n:127.0.0.1:57619_    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 281943 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:38221_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_A7B429E85CFE5361-001/shard-2-001/cores
   [junit4]   2> 281944 INFO  
(TEST-SharedFSAutoReplicaFailoverTest.test-seed#[A7B429E85CFE5361]) 
[n:127.0.0.1:38221_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 281946 INFO  
(OverseerStateUpdate-97306034565873668-127.0.0.1:57313_-n_0000000000) 
[n:127.0.0.1:57313_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard 
shard=shard1
   [junit4]   2> 282063 INFO  
(zkCallback-876-thread-1-processing-n:127.0.0.1:38221_) [n:127.0.0.1:38221_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 282063 INFO  
(zkCallback-870-thread-1-processing-n:127.0.0.1:57619_) [n:127.0.0.1:57619_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 282959 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:38221_) 
[n:127.0.0.1:38221_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using 
Lucene MatchVersion: 6.4.0
   [junit4]   2> 282991 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:38221_) 
[n:127.0.0.1:38221_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 283215 WARN  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:38221_) 
[n:127.0.0.1:38221_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] default search field in schema is text. WARNING: Deprecated, 
please use 'df' on request instead.
   [junit4]   2> 283219 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:38221_) 
[n:127.0.0.1:38221_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded 
schema test/1.6 with uniqueid field id
   [junit4]   2> 283288 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:38221_) 
[n:127.0.0.1:38221_ c:collection1   x:collection1] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 283288 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:38221_) 
[n:127.0.0.1:38221_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:50553/solr_hdfs_home
   [junit4]   2> 283288 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:38221_) 
[n:127.0.0.1:38221_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory 
Solr Kerberos Authentication disabled
   [junit4]   2> 283288 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:38221_) 
[n:127.0.0.1:38221_ c:co

[...truncated too long message...]

llection/state.json] for collection [target_collection] has occurred - 
updating... (live nodes size: [1])
   [junit4]   2> 4152093 INFO  
(zkCallback-3942-thread-5-processing-n:127.0.0.1:55765__%2Fk) 
[n:127.0.0.1:55765__%2Fk    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/target_collection/state.json] for collection 
[target_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4154054 WARN  
(zkCallback-3935-thread-6-processing-n:127.0.0.1:45504__%2Fk) 
[n:127.0.0.1:45504__%2Fk c:target_collection s:shard1 r:core_node3 
x:target_collection_shard1_replica2] o.a.s.c.SyncStrategy Closed, skipping sync 
up.
   [junit4]   2> 4154054 INFO  
(zkCallback-3935-thread-6-processing-n:127.0.0.1:45504__%2Fk) 
[n:127.0.0.1:45504__%2Fk c:target_collection s:shard1 r:core_node3 
x:target_collection_shard1_replica2] o.a.s.c.SolrCore 
[target_collection_shard1_replica2]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@2a89fe81
   [junit4]   2> 4154054 INFO  
(zkCallback-3935-thread-6-processing-n:127.0.0.1:45504__%2Fk) 
[n:127.0.0.1:45504__%2Fk c:target_collection s:shard1 r:core_node3 
x:target_collection_shard1_replica2] o.a.s.h.CdcrRequestHandler Solr core is 
being closed - shutting down CDCR handler @ target_collection:shard1
   [junit4]   2> 4154088 INFO  
(zkCallback-3935-thread-6-processing-n:127.0.0.1:45504__%2Fk) 
[n:127.0.0.1:45504__%2Fk c:target_collection s:shard1 r:core_node3 
x:target_collection_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric 
reporters for: solr.core.target_collection.shard1.replica2
   [junit4]   2> 4154088 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[A7B429E85CFE5361])
 [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 4154089 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[A7B429E85CFE5361])
 [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@23ce714a{SSL,[ssl, 
http/1.1]}{127.0.0.1:0}
   [junit4]   2> 4154089 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[A7B429E85CFE5361])
 [    ] o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@161e99e0{/_/k,null,UNAVAILABLE}
   [junit4]   2> 4154090 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[A7B429E85CFE5361])
 [    ] o.a.s.c.ChaosMonkey monkey: stop shard! 55765
   [junit4]   2> 4154090 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[A7B429E85CFE5361])
 [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1556685878
   [junit4]   2> 4154094 INFO  (coreCloseExecutor-7075-thread-1) 
[n:127.0.0.1:55765__%2Fk c:target_collection s:shard2 r:core_node4 
x:target_collection_shard2_replica2] o.a.s.c.SolrCore 
[target_collection_shard2_replica2]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@3a280c1f
   [junit4]   2> 4154095 INFO  (coreCloseExecutor-7075-thread-1) 
[n:127.0.0.1:55765__%2Fk c:target_collection s:shard2 r:core_node4 
x:target_collection_shard2_replica2] o.a.s.h.CdcrRequestHandler Solr core is 
being closed - shutting down CDCR handler @ target_collection:shard2
   [junit4]   2> 4154118 INFO  (coreCloseExecutor-7075-thread-1) 
[n:127.0.0.1:55765__%2Fk c:target_collection s:shard2 r:core_node4 
x:target_collection_shard2_replica2] o.a.s.m.SolrMetricManager Closing metric 
reporters for: solr.core.target_collection.shard2.replica2
   [junit4]   2> 4154119 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[A7B429E85CFE5361])
 [    ] o.a.s.c.Overseer Overseer 
(id=97306268703260705-127.0.0.1:55765__%2Fk-n_0000000007) closing
   [junit4]   2> 4154119 INFO  
(OverseerStateUpdate-97306268703260705-127.0.0.1:55765__%2Fk-n_0000000007) 
[n:127.0.0.1:55765__%2Fk    ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:55765__%2Fk
   [junit4]   2> 4155621 WARN  
(zkCallback-3942-thread-9-processing-n:127.0.0.1:55765__%2Fk) 
[n:127.0.0.1:55765__%2Fk    ] o.a.s.c.c.ZkStateReader ZooKeeper watch 
triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for 
/live_nodes]
   [junit4]   2> 4155622 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[A7B429E85CFE5361])
 [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 4155623 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[A7B429E85CFE5361])
 [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@45f0989a{SSL,[ssl, 
http/1.1]}{127.0.0.1:0}
   [junit4]   2> 4155623 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[A7B429E85CFE5361])
 [    ] o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@169b179e{/_/k,null,UNAVAILABLE}
   [junit4]   2> 4155625 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[A7B429E85CFE5361])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testReplicationAfterRestart
   [junit4]   2> 4155627 INFO  
(TEST-CdcrReplicationDistributedZkTest.testReplicationAfterRestart-seed#[A7B429E85CFE5361])
 [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34438 34438
   [junit4]   2> 4155738 INFO  (Thread-107604) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:34438 34438
   [junit4]   2> 4155740 WARN  (Thread-107604) [    ] o.a.s.c.ZkTestServer 
Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        35      /solr/aliases.json
   [junit4]   2>        18      /solr/configs/conf1
   [junit4]   2>        10      /solr/security.json
   [junit4]   2>        8       /solr/collections/tmp_collection/state.json
   [junit4]   2>        5       
/solr/collections/source_collection/leaders/shard1/leader
   [junit4]   2>        2       
/solr/collections/source_collection/leaders/shard2/leader
   [junit4]   2>        2       
/solr/collections/tmp_collection/leaders/shard2/leader
   [junit4]   2>        2       
/solr/collections/target_collection/leaders/shard2/leader
   [junit4]   2>        2       
/solr/collections/tmp_collection/leaders/shard1/leader
   [junit4]   2>        2       
/solr/collections/target_collection/leaders/shard1/leader
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        35      /solr/clusterprops.json
   [junit4]   2>        35      /solr/clusterstate.json
   [junit4]   2>        9       
/solr/collections/source_collection/cdcr/state/process
   [junit4]   2>        8       
/solr/collections/tmp_collection/cdcr/state/process
   [junit4]   2>        8       
/solr/collections/source_collection/cdcr/state/buffer
   [junit4]   2>        8       
/solr/collections/tmp_collection/cdcr/state/buffer
   [junit4]   2>        6       /solr/collections/source_collection/state.json
   [junit4]   2>        4       
/solr/collections/target_collection/cdcr/state/buffer
   [junit4]   2>        4       
/solr/collections/target_collection/cdcr/state/process
   [junit4]   2>        4       /solr/collections/target_collection/state.json
   [junit4]   2>        3       
/solr/overseer_elect/election/97306268703260680-127.0.0.1:39536__%2Fk-n_0000000001
   [junit4]   2>        2       
/solr/collections/source_collection/leader_elect/shard1/election/97306268703260686-core_node1-n_0000000000
   [junit4]   2>        2       
/solr/overseer_elect/election/97306268703260705-127.0.0.1:55765__%2Fk-n_0000000007
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        305     /solr/overseer/collection-queue-work
   [junit4]   2>        87      /solr/overseer/queue
   [junit4]   2>        35      /solr/collections
   [junit4]   2>        33      /solr/live_nodes
   [junit4]   2>        28      /solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> 4155741 INFO  
(SUITE-CdcrReplicationDistributedZkTest-seed#[A7B429E85CFE5361]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.4/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_A7B429E85CFE5361-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
{_version_=PostingsFormat(name=Asserting), id=BlockTreeOrds(blocksize=128)}, 
docValues:{}, maxPointsInLeafNode=1014, maxMBSortInHeap=7.103990925602041, 
sim=RandomSimilarity(queryNorm=true,coord=no): {}, 
locale=ja-JP-u-ca-japanese-x-lvariant-JP, timezone=Antarctica/Davis
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 
1.8.0_102 (64-bit)/cpus=4,threads=1,free=233252368,total=524812288
   [junit4]   2> NOTE: All tests run in this JVM: [ExternalFileFieldSortTest, 
TestEmbeddedSolrServerConstructors, DistributedQueryElevationComponentTest, 
TestFastWriter, TestAtomicUpdateErrorCases, SyncSliceTest, 
TestValueSourceCache, TestAuthorizationFramework, TestScoreJoinQPNoScore, 
BasicDistributedZkTest, TestFieldCache, 
PeerSyncWithIndexFingerprintCachingTest, PreAnalyzedUpdateProcessorTest, 
ChaosMonkeyNothingIsSafeTest, TestHashQParserPlugin, 
TestLocalFSCloudBackupRestore, DistributedDebugComponentTest, TestBlobHandler, 
RollingRestartTest, DateFieldTest, TestSortByMinMaxFunction, 
HdfsWriteToMultipleCollectionsTest, TestReplicationHandler, TestCloudRecovery, 
TestRealTimeGet, TestLeaderElectionWithEmptyReplica, 
TestComplexPhraseLeadingWildcard, DistribDocExpirationUpdateProcessorTest, 
HdfsCollectionsAPIDistributedZkTest, HdfsBasicDistributedZkTest, 
DistributedFacetPivotSmallTest, TestSlowCompositeReaderWrapper, TestLazyCores, 
TestBadConfig, TestFaceting, BasicZkTest, 
TestPerFieldSimilarityWithDefaultOverride, RuleEngineTest, TestDocumentBuilder, 
CdcrVersionReplicationTest, TestJmxIntegration, TestIntervalFaceting, 
TestReplicaProperties, TestCustomDocTransformer, 
ConjunctionSolrSpellCheckerTest, TestNumericTerms64, TestOmitPositions, 
JSONWriterTest, TestSha256AuthenticationProvider, HLLUtilTest, 
TestStressRecovery, DeleteLastCustomShardedReplicaTest, 
TestReRankQParserPlugin, QueryEqualityTest, MigrateRouteKeyTest, 
CdcrReplicationDistributedZkTest]
   [junit4] Completed [543/680 (5!)] on J0 in 1621.03s, 11 tests, 1 failure <<< 
FAILURES!

[...truncated 62900 lines...]



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org