Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/761/

4 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
KeeperErrorCode = Session expired for /clusterstate.json

Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = 
Session expired for /clusterstate.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
        at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
        at 
org.apache.solr.common.cloud.ZkStateReader.refreshLegacyClusterState(ZkStateReader.java:462)
        at 
org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:256)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testNoCollectionSpecified(CollectionsAPIDistributedZkTest.java:465)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:170)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.search.TestReloadDeadlock.testReloadDeadlock

Error Message:
Captured an uncaught exception in thread: Thread[id=63936, name=WRITER0, 
state=RUNNABLE, group=TGRP-TestReloadDeadlock]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=63936, name=WRITER0, state=RUNNABLE, 
group=TGRP-TestReloadDeadlock]
Caused by: java.lang.RuntimeException: org.apache.solr.common.SolrException: 
Error opening new searcher
        at __randomizedtesting.SeedInfo.seed([2669A7F1886DBA36]:0)
        at 
org.apache.solr.search.TestReloadDeadlock$1.run(TestReloadDeadlock.java:166)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1673)
        at org.apache.solr.core.SolrCore.getRealtimeSearcher(SolrCore.java:1530)
        at 
org.apache.solr.update.VersionInfo.getVersionFromIndex(VersionInfo.java:202)
        at org.apache.solr.update.UpdateLog.lookupVersion(UpdateLog.java:783)
        at 
org.apache.solr.update.VersionInfo.lookupVersion(VersionInfo.java:195)
        at 
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1088)
        at 
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:705)
        at 
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
        at 
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleAdds(JsonLoader.java:470)
        at 
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:134)
        at 
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:113)
        at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:76)
        at 
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:98)
        at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2079)
        at 
org.apache.solr.servlet.DirectSolrConnection.request(DirectSolrConnection.java:131)
        at org.apache.solr.SolrTestCaseJ4.updateJ(SolrTestCaseJ4.java:1104)
        at 
org.apache.solr.SolrTestCaseJ4.addAndGetVersion(SolrTestCaseJ4.java:1250)
        at 
org.apache.solr.search.TestReloadDeadlock.addDoc(TestReloadDeadlock.java:200)
        at 
org.apache.solr.search.TestReloadDeadlock.access$100(TestReloadDeadlock.java:46)
        at 
org.apache.solr.search.TestReloadDeadlock$1.run(TestReloadDeadlock.java:156)
Caused by: java.lang.NullPointerException
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1642)
        ... 21 more


FAILED:  org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTests

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 
seconds
        at 
__randomizedtesting.SeedInfo.seed([2669A7F1886DBA36:2E09D2DD8763923D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:172)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:133)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:128)
        at 
org.apache.solr.cloud.BaseCdcrDistributedZkTest.waitForRecoveriesToFinish(BaseCdcrDistributedZkTest.java:465)
        at 
org.apache.solr.cloud.BaseCdcrDistributedZkTest.clearTargetCollection(BaseCdcrDistributedZkTest.java:343)
        at 
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTestOps(CdcrReplicationDistributedZkTest.java:431)
        at 
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTests(CdcrReplicationDistributedZkTest.java:54)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=20263, name=collection1, 
state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=20263, name=collection1, state=RUNNABLE, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:51261/_: Could not find collection : 
awholynewstresscollection_collection1_0
        at __randomizedtesting.SeedInfo.seed([2669A7F1886DBA36]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:857)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:800)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:895)




Build Log:
[...truncated 10648 lines...]
   [junit4] Suite: org.apache.solr.cloud.CdcrReplicationDistributedZkTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/init-core-data-001
   [junit4]   2> 1213554 INFO  
(SUITE-CdcrReplicationDistributedZkTest-seed#[2669A7F1886DBA36]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1213554 INFO  
(SUITE-CdcrReplicationDistributedZkTest-seed#[2669A7F1886DBA36]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /x_/fn
   [junit4]   2> 1213557 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1213557 INFO  (Thread-3455) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1213557 INFO  (Thread-3455) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 1213657 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:46345
   [junit4]   2> 1213657 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1213658 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1213660 INFO  (zkCallback-1253-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@5979d463 
name:ZooKeeperConnection Watcher:127.0.0.1:46345 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1213660 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1213660 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1213660 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1213665 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x14f184da9bb0000, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1213665 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1213667 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1213668 INFO  (zkCallback-1254-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@10e869be 
name:ZooKeeperConnection Watcher:127.0.0.1:46345/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1213668 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1213668 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1213668 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1213670 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1213671 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1213672 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1213673 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-cdcr.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 1213674 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1213676 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 1213676 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1213678 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1213678 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1213679 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 1213679 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1213681 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 1213681 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1213682 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 1213683 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1213684 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1213684 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1213686 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1213686 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1213688 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1213688 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1213689 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1213690 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1213691 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 1213691 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1213695 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting doTests
   [junit4]   2> 1214024 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores/collection1
   [junit4]   2> 1214026 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 1214032 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@5c0bb2d7{/x_/fn,null,AVAILABLE}
   [junit4]   2> 1214033 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@1b73e5be{HTTP/1.1}{127.0.0.1:35787}
   [junit4]   2> 1214033 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.e.j.s.Server Started @1217276ms
   [junit4]   2> 1214033 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {shards=shard1, 
hostContext=/x_/fn, hostPort=35787, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores}
   [junit4]   2> 1214033 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1214033 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/'
   [junit4]   2> 1214049 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1214049 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1214051 INFO  (zkCallback-1255-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@5af420a9 
name:ZooKeeperConnection Watcher:127.0.0.1:46345/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1214051 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1214051 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1214053 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1214053 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/solr.xml
   [junit4]   2> 1214058 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores
   [junit4]   2> 1214058 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.CoreContainer New CoreContainer 1717945705
   [junit4]   2> 1214058 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/]
   [junit4]   2> 1214058 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/lib
   [junit4]   2> 1214058 WARN  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/lib).
   [junit4]   2> 1214066 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme 
: ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 
10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 
5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1214067 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1214067 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1214068 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1214068 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46345/solr
   [junit4]   2> 1214068 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1214068 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1214069 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1214070 INFO  (zkCallback-1257-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@46d9db0a 
name:ZooKeeperConnection Watcher:127.0.0.1:46345 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1214070 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1214071 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1214073 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ConnectionManager Waiting for client 
to connect to ZooKeeper
   [junit4]   2> 1214074 INFO  
(zkCallback-1258-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@2ba8f26a 
name:ZooKeeperConnection Watcher:127.0.0.1:46345/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1214074 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ConnectionManager Client is connected 
to ZooKeeper
   [junit4]   2> 1214075 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1214078 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-queue-work
   [junit4]   2> 1214080 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-running
   [junit4]   2> 1214082 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-completed
   [junit4]   2> 1214084 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-failure
   [junit4]   2> 1214086 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1214088 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1214090 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: 
/clusterstate.json
   [junit4]   2> 1214099 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1214104 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:35787_x_%2Ffn
   [junit4]   2> 1214104 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:35787_x_%2Ffn
   [junit4]   2> 1214106 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1214108 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/election
   [junit4]   2> 1214109 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1214110 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.LeaderElector Joined leadership 
election with path: 
/overseer_elect/election/94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000
   [junit4]   2> 1214110 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.OverseerElectionContext I am going to 
be the leader 127.0.0.1:35787_x_%2Ffn
   [junit4]   2> 1214110 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/leader
   [junit4]   2> 1214112 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.Overseer Overseer 
(id=94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) starting
   [junit4]   2> 1214113 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/queue-work
   [junit4]   2> 1214118 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.OverseerAutoReplicaFailoverThread 
Starting OverseerAutoReplicaFailoverThread 
autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=30000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1214122 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.Overseer Starting to work on the main 
queue
   [junit4]   2> 1214123 INFO  
(OverseerCollectionProcessor-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000)
 [n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.OverseerProcessor Process current 
queue of overseer operations
   [junit4]   2> 1214123 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ZkStateReader Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 1214131 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ZkStateReader Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 1214150 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.CoreContainer Security conf doesn't 
exist. Skipping setup for authorization module.
   [junit4]   2> 1214150 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.CoreContainer No authentication plugin 
used.
   [junit4]   2> 1214151 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.CoresLocator Looking for core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores
   [junit4]   2> 1214152 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores/collection1,
 collection=tmp_collection, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores/collection1/,
 coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1214152 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.CoresLocator Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores/collection1/
   [junit4]   2> 1214153 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 1214155 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1214156 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ZkController numShards not found on descriptor - reading it from system 
property
   [junit4]   2> 1214156 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1214159 INFO  
(zkCallback-1258-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.DistributedQueue NodeChildrenChanged 
fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1214160 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1214160 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.s.SolrDispatchFilter 
SolrDispatchFilter.init() done
   [junit4]   2> 1214160 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:35787/x_/fn";,
   [junit4]   2>   "node_name":"127.0.0.1:35787_x_%2Ffn",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"tmp_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1214160 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.o.ReplicaMutator Update state 
numShards=2 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:35787/x_/fn";,
   [junit4]   2>   "node_name":"127.0.0.1:35787_x_%2Ffn",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"tmp_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1214161 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.o.ClusterStateMutator building a new 
cName: tmp_collection
   [junit4]   2> 1214161 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.o.ReplicaMutator Assigning new node to 
shard shard=shard2
   [junit4]   2> 1214162 INFO  
(zkCallback-1258-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1214162 INFO  
(zkCallback-1258-thread-2-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1215156 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1215157 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ZkController Check for collection zkNode:tmp_collection
   [junit4]   2> 1215157 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ZkController Creating collection in ZooKeeper:tmp_collection
   [junit4]   2> 1215157 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ZkController Looking for collection configName
   [junit4]   2> 1215158 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ZkController Only one config set found in zk - using it:conf1
   [junit4]   2> 1215158 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.c.SolrZkClient makePath: /collections/tmp_collection
   [junit4]   2> 1215159 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config from:/collections/tmp_collection
   [junit4]   2> 1215160 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.c.ZkStateReader path=/collections/tmp_collection configName=conf1 
specified config exists in ZooKeeper
   [junit4]   2> 1215160 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores/collection1/'
   [junit4]   2> 1215172 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] o.a.s.c.Config 
loaded config solrconfig.xml with version 0 
   [junit4]   2> 1215178 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] o.a.s.c.SolrConfig 
current version of requestparams : -1
   [junit4]   2> 1215181 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1215188 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] o.a.s.c.Config 
Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1215191 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1215195 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1215285 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1215286 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1215287 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1215290 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1215309 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default 
properties: Can't find resource 'configsetprops.json' in classpath or 
'/configs/conf1', 
cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1215309 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from 
collection tmp_collection
   [junit4]   2> 1215309 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1215309 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores/collection1/],
 dataDir=[null]
   [junit4]   2> 1215310 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr 
mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45423ff8
   [junit4]   2> 1215310 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores/collection1/data
   [junit4]   2> 1215310 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore New index directory detected: old=null 
new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores/collection1/data/index/
   [junit4]   2> 1215310 WARN  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores/collection1/data/index'
 doesn't exist. Creating new index...
   [junit4]   2> 1215311 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores/collection1/data/index
   [junit4]   2> 1215311 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: 
minMergeSize=0, mergeFactor=10, maxMergeSize=2000300610, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.1]
   [junit4]   2> 1215312 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>        commit{dir=MockDirectoryWrapper(RAMDirectory@713482df 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6af45d86),segFN=segments_1,generation=1}
   [junit4]   2> 1215312 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 1215312 INFO  
(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:35787_x_%2Ffn 
c:tmp_collection s:shard2 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking 
for old index directories to cleanup for core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores/collection1/data/
   [junit4]   2> 1215313 WARN  
(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:35787_x_%2Ffn 
c:tmp_collection s:shard2 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-001/cores/collection1/data/
 does not point to a valid data directory; skipping clean-up of old index 
directories.
   [junit4]   2> 1215318 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "cdcr-processor-chain"
   [junit4]   2> 1215332 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as 
default, creating implicit default
   [junit4]   2> 1215337 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1215338 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1215339 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1215340 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1215349 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.RequestHandlers Registered paths: 
/admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
   [junit4]   2> 1215350 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore Using default statsCache cache: 
org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1215360 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 1215360 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= 
defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 
numVersionBuckets=65536
   [junit4]   2> 1215361 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 1215361 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 1215361 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=38, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=15.8701171875, 
floorSegmentMB=0.7890625, forceMergeDeletesPctAllowed=11.28749836845501, 
segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1215362 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>        commit{dir=MockDirectoryWrapper(RAMDirectory@713482df 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6af45d86),segFN=segments_1,generation=1}
   [junit4]   2> 1215362 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 1215362 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@7e64d415[collection1] 
main
   [junit4]   2> 1215362 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/tmp_collection
   [junit4]   2> 1215367 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.c.ZkStateReader path=/collections/tmp_collection 
configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1215367 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based 
storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1215367 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO 
with znodeBase: /configs/conf1
   [junit4]   2> 1215368 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1215368 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1215368 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode 
/configs/conf1/_rest_managed.json
   [junit4]   2> 1215369 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path 
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1215369 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1215369 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1215373 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.c.SolrZkClient makePath: 
/collections/tmp_collection/cdcr/state
   [junit4]   2> 1215376 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.h.CdcrBufferStateManager Created znode 
/collections/tmp_collection/cdcr/state/buffer
   [junit4]   2> 1215388 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.h.CdcrProcessStateManager Created znode 
/collections/tmp_collection/cdcr/state/process
   [junit4]   2> 1215404 INFO  
(searcherExecutor-2315-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@7e64d415[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1215405 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed 
version buckets
   [junit4]   2> 1215405 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 
65536 version buckets from index
   [junit4]   2> 1215405 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed 
version bucket highest value from index
   [junit4]   2> 1215405 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent 
updates, using new clock 1509133426532286464
   [junit4]   2> 1215407 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with 
highest version 1509133426532286464
   [junit4]   2> 1215407 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1215408 INFO  
(coreLoadExecutor-2314-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1215408 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.ZkController Register replica - core:collection1 
address:http://127.0.0.1:35787/x_/fn collection:tmp_collection shard:shard2
   [junit4]   2> 1215410 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.c.SolrZkClient makePath: 
/collections/tmp_collection/leader_elect/shard2/election
   [junit4]   2> 1215413 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: 
/collections/tmp_collection/leader_elect/shard2/election/94320839037222916-core_node1-n_0000000000
   [junit4]   2> 1215414 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process 
for shard shard2
   [junit4]   2> 1215415 INFO  
(zkCallback-1258-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.DistributedQueue NodeChildrenChanged 
fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1215415 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to 
continue.
   [junit4]   2> 1215415 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try 
and sync
   [junit4]   2> 1215415 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:35787/x_/fn/collection1/
   [junit4]   2> 1215416 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1215416 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:35787/x_/fn/collection1/ 
has no replicas
   [junit4]   2> 1215416 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:35787/x_/fn/collection1/ shard2
   [junit4]   2> 1215416 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.c.SolrZkClient makePath: 
/collections/tmp_collection/leaders/shard2
   [junit4]   2> 1215419 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"tmp_collection"} current state version: 1
   [junit4]   2> 1215436 INFO  
(zkCallback-1258-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.h.CdcrProcessStateManager Received new 
leader state @ tmp_collection:shard2
   [junit4]   2> 1215439 INFO  
(zkCallback-1258-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.DistributedQueue NodeChildrenChanged 
fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1215440 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"tmp_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:35787/x_/fn";,
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 1215546 INFO  
(zkCallback-1258-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1215546 INFO  
(zkCallback-1258-thread-2-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1215591 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.ZkController We are 
http://127.0.0.1:35787/x_/fn/collection1/ and leader is 
http://127.0.0.1:35787/x_/fn/collection1/
   [junit4]   2> 1215591 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 
baseURL=http://127.0.0.1:35787/x_/fn
   [junit4]   2> 1215591 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1215591 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1215591 INFO  
(coreZkRegister-2308-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn 
x:collection1 s:shard2 c:tmp_collection r:core_node1) 
[n:127.0.0.1:35787_x_%2Ffn c:tmp_collection s:shard2 r:core_node1 
x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading 
it from system property
   [junit4]   2> 1215592 INFO  
(zkCallback-1258-thread-2-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.DistributedQueue NodeChildrenChanged 
fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1215593 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:35787/x_/fn";,
   [junit4]   2>   "node_name":"127.0.0.1:35787_x_%2Ffn",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"tmp_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 1215594 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.o.ReplicaMutator Update state 
numShards=2 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:35787/x_/fn";,
   [junit4]   2>   "node_name":"127.0.0.1:35787_x_%2Ffn",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"tmp_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1215696 INFO  
(zkCallback-1258-thread-2-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1215696 INFO  
(zkCallback-1258-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1215985 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-002/cores/collection1
   [junit4]   2> 1215987 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 1215988 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@36872ae3{/x_/fn,null,AVAILABLE}
   [junit4]   2> 1215989 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@30ddf1ce{HTTP/1.1}{127.0.0.1:42647}
   [junit4]   2> 1215989 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.e.j.s.Server Started @1219232ms
   [junit4]   2> 1215989 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {shards=shard2, 
hostContext=/x_/fn, hostPort=42647, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-002/cores}
   [junit4]   2> 1215990 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1215990 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-002/'
   [junit4]   2> 1216005 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1216005 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1216007 INFO  (zkCallback-1259-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@3d30e940 
name:ZooKeeperConnection Watcher:127.0.0.1:46345/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1216007 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1216007 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1216009 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1216009 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-002/solr.xml
   [junit4]   2> 1216014 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-002/cores
   [junit4]   2> 1216015 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.CoreContainer New CoreContainer 416371356
   [junit4]   2> 1216015 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-002/]
   [junit4]   2> 1216015 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-002/lib
   [junit4]   2> 1216015 WARN  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-002/lib).
   [junit4]   2> 1216022 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme 
: ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 
10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 
5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1216024 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1216024 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1216025 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1216025 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46345/solr
   [junit4]   2> 1216025 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1216025 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1216028 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1216028 INFO  (zkCallback-1261-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@9c88ba name:ZooKeeperConnection 
Watcher:127.0.0.1:46345 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 1216029 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1216029 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1216031 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.c.ConnectionManager Waiting for client 
to connect to ZooKeeper
   [junit4]   2> 1216032 INFO  
(zkCallback-1262-thread-1-processing-n:127.0.0.1:42647_x_%2Ffn) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@26fa671a 
name:ZooKeeperConnection Watcher:127.0.0.1:46345/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1216032 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.c.ConnectionManager Client is connected 
to ZooKeeper
   [junit4]   2> 1216036 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.c.ZkStateReader Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 1217041 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:42647_x_%2Ffn
   [junit4]   2> 1217042 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:42647_x_%2Ffn
   [junit4]   2> 1217044 INFO  
(zkCallback-1258-thread-2-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ZkStateReader A live node change: 
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has 
occurred - updating... (live nodes size: 1)
   [junit4]   2> 1217044 INFO  
(zkCallback-1258-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.c.ZkStateReader A live node change: 
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has 
occurred - updating... (live nodes size: 1)
   [junit4]   2> 1217044 INFO  
(zkCallback-1262-thread-1-processing-n:127.0.0.1:42647_x_%2Ffn) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.c.ZkStateReader A live node change: 
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has 
occurred - updating... (live nodes size: 1)
   [junit4]   2> 1217045 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1217046 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.LeaderElector Joined leadership 
election with path: 
/overseer_elect/election/94320839037222919-127.0.0.1:42647_x_%2Ffn-n_0000000001
   [junit4]   2> 1217047 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.LeaderElector Watching path 
/overseer_elect/election/94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000 
to know if I could be the leader
   [junit4]   2> 1217059 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.CoreContainer Security conf doesn't 
exist. Skipping setup for authorization module.
   [junit4]   2> 1217059 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.CoreContainer No authentication plugin 
used.
   [junit4]   2> 1217060 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.CoresLocator Looking for core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-002/cores
   [junit4]   2> 1217061 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-002/cores/collection1,
 collection=tmp_collection, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-002/cores/collection1/,
 coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1217061 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.CoresLocator Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest_2669A7F1886DBA36-001/jetty-002/cores/collection1/
   [junit4]   2> 1217061 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 1217062 INFO  
(coreLoadExecutor-2326-thread-1-processing-n:127.0.0.1:42647_x_%2Ffn) 
[n:127.0.0.1:42647_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1217062 INFO  
(coreLoadExecutor-2326-thread-1-processing-n:127.0.0.1:42647_x_%2Ffn) 
[n:127.0.0.1:42647_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ZkController numShards not found on descriptor - reading it from system 
property
   [junit4]   2> 1217062 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1217062 INFO  
(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[2669A7F1886DBA36]) 
[n:127.0.0.1:42647_x_%2Ffn    ] o.a.s.s.SolrDispatchFilter 
SolrDispatchFilter.init() done
   [junit4]   2> 1217063 INFO  
(coreLoadExecutor-2326-thread-1-processing-n:127.0.0.1:42647_x_%2Ffn) 
[n:127.0.0.1:42647_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ZkController Registering watch for collection tmp_collection
   [junit4]   2> 1217063 INFO  
(coreLoadExecutor-2326-thread-1-processing-n:127.0.0.1:42647_x_%2Ffn) 
[n:127.0.0.1:42647_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.c.ZkStateReader addZkWatch tmp_collection
   [junit4]   2> 1217063 INFO  
(zkCallback-1258-thread-1-processing-n:127.0.0.1:35787_x_%2Ffn) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.DistributedQueue NodeChildrenChanged 
fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1217063 INFO  
(coreLoadExecutor-2326-thread-1-processing-n:127.0.0.1:42647_x_%2Ffn) 
[n:127.0.0.1:42647_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.c.ZkStateReader Deleting data for tmp_collection
   [junit4]   2> 1217064 INFO  
(coreLoadExecutor-2326-thread-1-processing-n:127.0.0.1:42647_x_%2Ffn) 
[n:127.0.0.1:42647_x_%2Ffn c:tmp_collection   x:collection1] 
o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1217064 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:42647/x_/fn";,
   [junit4]   2>   "node_name":"127.0.0.1:42647_x_%2Ffn",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"tmp_collection",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 1217064 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.o.ReplicaMutator Update state 
numShards=2 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:42647/x_/fn";,
   [junit4]   2>   "node_name":"127.0.0.1:42647_x_%2Ffn",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"tmp_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1217064 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn    ] o.a.s.c.o.ReplicaMutator Collection already 
exists with numShards=2
   [junit4]   2> 1217064 INFO  
(OverseerStateUpdate-94320839037222916-127.0.0.1:35787_x_%2Ffn-n_0000000000) 
[n:127.0.0.1:35787_x_%2Ffn

[...truncated too long message...]

ter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2>        at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:227)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:212)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:461)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:212)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:592)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:145)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:822)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1720)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1699)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:197)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.teardownClass(HdfsCollectionsAPIDistributedZkTest.java:50)
   [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:497)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2>        ... 53 more
   [junit4]   2> 4813205 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[2669A7F1886DBA36]-worker) 
[n:127.0.0.1:42355__ c:addReplicaColl s:shard2 r:core_node6 
x:addReplicaColl_shard2_replica3] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Aug 10, 2015 12:49:47 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=CheapBastard, 
sim=DefaultSimilarity, locale=be, timezone=NET
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 
1.8.0_45 (64-bit)/cpus=4,threads=2,free=235471840,total=535298048
   [junit4]   2> NOTE: All tests run in this JVM: 
[LeaderElectionIntegrationTest, HdfsNNFailoverTest, 
UpdateRequestProcessorFactoryTest, TestReplicationHandlerBackup, 
ClusterStateTest, RequestHandlersTest, TestMinMaxOnMultiValuedField, TestJoin, 
TestManagedSchemaFieldResource, TestPhraseSuggestions, TestRemoteStreaming, 
TestConfigSetProperties, TimeZoneUtilsTest, PeerSyncTest, DebugComponentTest, 
TestMergePolicyConfig, SuggesterWFSTTest, TestSha256AuthenticationProvider, 
BasicDistributedZkTest, DeleteInactiveReplicaTest, MergeStrategyTest, 
TestRangeQuery, DateFieldTest, TestFieldTypeCollectionResource, 
CurrencyFieldXmlFileTest, TestArbitraryIndexDir, TestSolrDeletionPolicy1, 
TestDFRSimilarityFactory, LeaderFailoverAfterPartitionTest, 
DocValuesMissingTest, DateMathParserTest, ExternalFileFieldSortTest, 
BJQParserTest, UUIDFieldTest, TestScoreJoinQPScore, TestFaceting, 
TestRawResponseWriter, BlockCacheTest, FastVectorHighlighterTest, 
SchemaVersionSpecificBehaviorTest, TestCodecSupport, CSVRequestHandlerTest, 
TestRealTimeGet, DistributedFacetPivotLargeTest, TestJmxIntegration, 
ZkStateReaderTest, TestCloudPivotFacet, TestHdfsUpdateLog, 
DeleteLastCustomShardedReplicaTest, BigEndianAscendingWordSerializerTest, 
HdfsDirectoryTest, CollectionReloadTest, PrimitiveFieldTypeTest, 
TestReversedWildcardFilterFactory, TestExactStatsCache, TestDocumentBuilder, 
FullHLLTest, TestUpdate, TestLRUStatsCache, SharedFSAutoReplicaFailoverTest, 
TestCloudManagedSchemaConcurrent, TestSchemaNameResource, 
DistributedFacetPivotSmallAdvancedTest, TestBlobHandler, MBeansHandlerTest, 
CdcrReplicationDistributedZkTest, TestFieldTypeResource, 
URLClassifyProcessorTest, CdcrUpdateLogTest, TestDefaultSearchFieldResource, 
TestConfig, TermVectorComponentTest, SignatureUpdateProcessorFactoryTest, 
DistributedDebugComponentTest, StandardRequestHandlerTest, 
TestSolrConfigHandlerCloud, CoreMergeIndexesAdminHandlerTest, TestMacros, 
TestMaxScoreQueryParser, TestNamedUpdateProcessors, TestBadConfig, 
RecoveryAfterSoftCommitTest, TestStressLucene, TestTrackingShardHandlerFactory, 
CollectionStateFormat2Test, SolrCoreTest, TestUniqueKeyFieldResource, 
DistributedQueryComponentCustomSortTest, IndexSchemaTest, 
BasicDistributedZk2Test, SyncSliceTest, ShardRoutingTest, BasicZkTest, 
RecoveryZkTest, TestReplicationHandler, CloudExitableDirectoryReaderTest, 
CollectionTooManyReplicasTest, DistributedQueueTest, 
DistributedVersionInfoTest, MultiThreadedOCPTest, OverseerStatusTest, 
ReplicationFactorTest, SaslZkACLProviderTest, ShardSplitTest, 
HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [540/540] on J0 in 913.68s, 1 test, 1 error <<< FAILURES!

[...truncated 14 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:723:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:660:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:230:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:514:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1429:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:984:
 There were test failures: 540 suites, 2127 tests, 3 errors, 1 failure, 26 
ignored (14 assumptions)

Total time: 199 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to