Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/839/

1 tests failed.
FAILED:  
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR

Error Message:
Captured an uncaught exception in thread: Thread[id=10668, 
name=coreZkRegister-1043-thread-2, state=RUNNABLE, 
group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=10668, name=coreZkRegister-1043-thread-2, 
state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]
Caused by: java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([23A6EC34FC14F77A]:0)
        at 
org.apache.solr.cloud.ZkController.updateLeaderInitiatedRecoveryState(ZkController.java:2126)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:433)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1113)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:926)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:881)
        at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10141 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/init-core-data-001
   [junit4]   2> 440301 INFO  
(SUITE-LeaderInitiatedRecoveryOnShardRestartTest-seed#[23A6EC34FC14F77A]-worker)
 [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system 
property: /f_dlr/h
   [junit4]   2> 440305 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 440309 INFO  (Thread-9093) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 440309 INFO  (Thread-9093) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 440409 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.ZkTestServer start zk server on port:34217
   [junit4]   2> 440409 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 440412 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 440420 INFO  (zkCallback-226-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@7c316a08 
name:ZooKeeperConnection Watcher:127.0.0.1:34217 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440420 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 440421 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 440421 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 440424 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 440432 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 440442 INFO  (zkCallback-227-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@189070cc 
name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440442 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 440443 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 440443 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 440444 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 440446 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 440447 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 440448 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 440448 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 440450 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 440450 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 440451 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 440452 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 440453 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 440453 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 440454 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 440454 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 440455 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 440455 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 440457 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 440457 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 440458 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 440458 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 440459 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 440460 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 440464 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 440464 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 440466 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 440466 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 440802 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1
   [junit4]   2> 440803 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 440805 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@6111d254{/f_dlr/h,null,AVAILABLE}
   [junit4]   2> 440806 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.e.j.s.ServerConnector Started 
ServerConnector@60c4ff2b{HTTP/1.1}{127.0.0.1:58984}
   [junit4]   2> 440806 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.e.j.s.Server Started @443587ms
   [junit4]   2> 440806 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/tempDir-001/control/data,
 hostContext=/f_dlr/h, hostPort=58984, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores}
   [junit4]   2> 440807 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 440807 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/'
   [junit4]   2> 440807 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr 
(NoInitialContextEx)
   [junit4]   2> 440807 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not 
find system property or JNDI)
   [junit4]   2> 440822 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 440824 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 440827 INFO  (zkCallback-228-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@3cf43ab3 
name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440827 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 440828 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 440830 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 440830 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/solr.xml
   [junit4]   2> 440835 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores
   [junit4]   2> 440835 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.CoreContainer New CoreContainer 1220789211
   [junit4]   2> 440835 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/]
   [junit4]   2> 440835 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/lib
   [junit4]   2> 440835 WARN  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/lib).
   [junit4]   2> 440844 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 
90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 
20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 
2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : 
false,useRetries : false,
   [junit4]   2> 440845 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with 
params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 440846 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 440846 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 440846 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34217/solr
   [junit4]   2> 440846 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 440846 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 440847 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 440855 INFO  (zkCallback-230-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@20510216 
name:ZooKeeperConnection Watcher:127.0.0.1:34217 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440855 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 440855 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 440859 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ConnectionManager Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 440881 INFO  
(zkCallback-231-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@12de497d 
name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440881 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ConnectionManager Client is 
connected to ZooKeeper
   [junit4]   2> 440882 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/queue
   [junit4]   2> 440889 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-queue-work
   [junit4]   2> 440891 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-running
   [junit4]   2> 440892 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-completed
   [junit4]   2> 440894 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-failure
   [junit4]   2> 440895 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 440901 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/aliases.json
   [junit4]   2> 440905 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/clusterstate.json
   [junit4]   2> 440906 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/security.json
   [junit4]   2> 440907 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.ZkController Register node as live 
in ZooKeeper:/live_nodes/127.0.0.1:58984_f_dlr%2Fh
   [junit4]   2> 440907 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:58984_f_dlr%2Fh
   [junit4]   2> 440909 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect
   [junit4]   2> 440910 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/election
   [junit4]   2> 440912 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 440913 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.LeaderElector Joined leadership 
election with path: 
/overseer_elect/election/94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000
   [junit4]   2> 440913 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.OverseerElectionContext I am going 
to be the leader 127.0.0.1:58984_f_dlr%2Fh
   [junit4]   2> 440913 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/leader
   [junit4]   2> 440914 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.Overseer Overseer 
(id=94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) starting
   [junit4]   2> 440921 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/queue-work
   [junit4]   2> 440929 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.OverseerAutoReplicaFailoverThread 
Starting OverseerAutoReplicaFailoverThread 
autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=30000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 440938 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.Overseer Starting to work on the main 
queue
   [junit4]   2> 440938 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ZkStateReader Updating cluster 
state from ZooKeeper... 
   [junit4]   2> 440948 INFO  
(OverseerCollectionConfigSetProcessor-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000)
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.OverseerTaskProcessor Process 
current queue of overseer operations
   [junit4]   2> 440957 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ZkStateReader Updating cluster 
state from ZooKeeper... 
   [junit4]   2> 440970 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.CoreContainer Security conf doesn't 
exist. Skipping setup for authorization module.
   [junit4]   2> 440970 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.CoreContainer No authentication 
plugin used.
   [junit4]   2> 440971 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.CoresLocator Looking for core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores
   [junit4]   2> 440971 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1,
 collection=control_collection, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/,
 coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 440972 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.CoresLocator Found core collection1 
in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/
   [junit4]   2> 440972 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 440977 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.ZkController publishing state=down
   [junit4]   2> 440978 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.ZkController numShards not found on descriptor - reading it from system 
property
   [junit4]   2> 440978 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 440978 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.s.SolrDispatchFilter 
SolrDispatchFilter.init() done
   [junit4]   2> 440979 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.Overseer processMessage: queueSize: 
1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:58984/f_dlr/h";,
   [junit4]   2>   "node_name":"127.0.0.1:58984_f_dlr%2Fh",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 440980 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.ZkController look for our core node name
   [junit4]   2> 440980 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.o.ReplicaMutator Update state 
numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:58984/f_dlr/h";,
   [junit4]   2>   "node_name":"127.0.0.1:58984_f_dlr%2Fh",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 440980 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.o.ClusterStateMutator building a new 
cName: control_collection
   [junit4]   2> 440981 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.o.ReplicaMutator Assigning new node 
to shard shard=shard1
   [junit4]   2> 440982 INFO  
(zkCallback-231-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ZkStateReader A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 440996 INFO  
(zkCallback-231-thread-2-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ZkStateReader A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 441980 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 441980 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 441981 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 441981 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 441982 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 
specified config exists in ZooKeeper
   [junit4]   2> 441982 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/'
   [junit4]   2> 441982 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 441982 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 441998 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 442002 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 442006 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 442014 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.Config Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 442015 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 442020 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 442123 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Initialized with 
rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 442129 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 442130 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 442140 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 442143 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 442147 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 442148 WARN  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 442148 WARN  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 442148 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 442149 WARN  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 442149 WARN  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 442150 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default 
properties: Can't find resource 'configsetprops.json' in classpath or 
'/configs/conf1', 
cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 442150 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection   x:collection1] 
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from 
collection control_collection
   [junit4]   2> 442150 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 442150 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/],
 dataDir=[null]
   [junit4]   2> 442150 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr 
mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@392b887
   [junit4]   2> 442157 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/data
   [junit4]   2> 442157 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore New index directory detected: old=null 
new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/data/index/
   [junit4]   2> 442157 WARN  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/data/index'
 doesn't exist. Creating new index...
   [junit4]   2> 442157 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/data/index
   [junit4]   2> 442158 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@43e442ff
   [junit4]   2> 442164 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>        
commit{dir=RawDirectoryWrapper(SimpleFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/index-SimpleFSDirectory-001
 
lockFactory=org.apache.lucene.store.NativeFSLockFactory@74948fa1),segFN=segments_1,generation=1}
   [junit4]   2> 442165 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 442165 INFO  
(OldIndexDirectoryCleanupThreadForCore-collection1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup 
for core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/data/
   [junit4]   2> 442165 WARN  
(OldIndexDirectoryCleanupThreadForCore-collection1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.DirectoryFactory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/data/
 does not point to a valid data directory; skipping clean-up of old index 
directories.
   [junit4]   2> 442169 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "nodistrib"
   [junit4]   2> 442169 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "dedupe"
   [junit4]   2> 442169 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 442169 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "stored_sig"
   [junit4]   2> 442169 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 442169 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 442170 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 442170 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 442170 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as 
default, creating implicit default
   [junit4]   2> 442172 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 442172 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 442173 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 442174 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 442177 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.RequestHandlers Registered paths: 
/admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 442178 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Using default statsCache cache: 
org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 442181 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 442181 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= 
defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 
numVersionBuckets=65536
   [junit4]   2> 442182 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 442182 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 442182 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: 
minMergeSize=1677721, mergeFactor=3, maxMergeSize=2147483648, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.2216796875, noCFSRatio=1.0]
   [junit4]   2> 442183 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>        
commit{dir=RawDirectoryWrapper(SimpleFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/index-SimpleFSDirectory-001
 
lockFactory=org.apache.lucene.store.NativeFSLockFactory@74948fa1),segFN=segments_1,generation=1}
   [junit4]   2> 442183 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 442183 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@255ab702[collection1] 
main
   [junit4]   2> 442184 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 442185 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection 
configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 442185 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based 
storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 442186 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO 
with znodeBase: /configs/conf1
   [junit4]   2> 442186 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 442186 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 442186 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode 
/configs/conf1/_rest_managed.json
   [junit4]   2> 442186 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path 
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 442186 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 442187 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 442187 INFO  
(searcherExecutor-1006-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@255ab702[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 442187 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed 
version buckets
   [junit4]   2> 442187 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 
65536 version buckets from index
   [junit4]   2> 442187 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed 
version bucket highest value from index
   [junit4]   2> 442187 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent 
updates, using new clock 1516634569662529536
   [junit4]   2> 442190 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with 
highest version 1516634569662529536
   [junit4]   2> 442190 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 442190 INFO  
(coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 442191 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ZkController Register replica - core:collection1 
address:http://127.0.0.1:58984/f_dlr/h collection:control_collection 
shard:shard1
   [junit4]   2> 442191 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.c.SolrZkClient makePath: 
/collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 442195 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: 
/collections/control_collection/leader_elect/shard1/election/94789660481486852-core_node1-n_0000000000
   [junit4]   2> 442195 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process 
for shard shard1
   [junit4]   2> 442197 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to 
continue.
   [junit4]   2> 442197 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try 
and sync
   [junit4]   2> 442197 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:58984/f_dlr/h/collection1/
   [junit4]   2> 442197 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 442197 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:58984/f_dlr/h/collection1/ 
has no replicas
   [junit4]   2> 442197 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:58984/f_dlr/h/collection1/ shard1
   [junit4]   2> 442197 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.Overseer processMessage: queueSize: 
1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 442197 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.c.SolrZkClient makePath: 
/collections/control_collection/leaders/shard1
   [junit4]   2> 442200 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader 
registration node
   [junit4]   2> 442202 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.Overseer processMessage: queueSize: 
1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:58984/f_dlr/h";,
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 442305 INFO  
(zkCallback-231-thread-2-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ZkStateReader A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 442305 INFO  
(zkCallback-231-thread-3-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ZkStateReader A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 442354 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ZkController We are 
http://127.0.0.1:58984/f_dlr/h/collection1/ and leader is 
http://127.0.0.1:58984/f_dlr/h/collection1/
   [junit4]   2> 442354 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 
baseURL=http://127.0.0.1:58984/f_dlr/h
   [junit4]   2> 442354 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 442354 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 442354 INFO  
(coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading 
it from system property
   [junit4]   2> 442356 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.Overseer processMessage: queueSize: 
1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:58984/f_dlr/h";,
   [junit4]   2>   "node_name":"127.0.0.1:58984_f_dlr%2Fh",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 442357 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.o.ReplicaMutator Update state 
numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:58984/f_dlr/h";,
   [junit4]   2>   "node_name":"127.0.0.1:58984_f_dlr%2Fh",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 442461 INFO  
(zkCallback-231-thread-3-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ZkStateReader A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 442461 INFO  
(zkCallback-231-thread-2-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ZkStateReader A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 442479 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 442480 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 442485 INFO  (zkCallback-233-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@89c49b9 name:ZooKeeperConnection 
Watcher:127.0.0.1:34217/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 442485 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 442485 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 442485 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 442487 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause 
connection loss:false
   [junit4]   2> 442487 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with 
stateFormat=2
   [junit4]   2> 442487 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 442488 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 442489 INFO  (zkCallback-234-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@3cfc14c9 
name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 442489 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 442490 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 442493 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.Overseer processMessage: queueSize: 
1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2"} current state version: 3
   [junit4]   2> 442493 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.o.ClusterStateMutator building a new 
cName: collection1
   [junit4]   2> 442596 INFO  
(OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.o.ZkStateWriter going to 
create_collection /collections/collection1/state.json
   [junit4]   2> 442808 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores/collection1
   [junit4]   2> 442809 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001
   [junit4]   2> 442810 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 442811 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@3bbc54ef{/f_dlr/h,null,AVAILABLE}
   [junit4]   2> 442812 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.e.j.s.ServerConnector Started 
ServerConnector@616216a3{HTTP/1.1}{127.0.0.1:52480}
   [junit4]   2> 442812 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.e.j.s.Server Started @445593ms
   [junit4]   2> 442812 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/tempDir-001/jetty1,
 solrconfig=solrconfig.xml, hostContext=/f_dlr/h, hostPort=52480, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores}
   [junit4]   2> 442812 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 442812 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/'
   [junit4]   2> 442813 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr 
(NoInitialContextEx)
   [junit4]   2> 442813 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not 
find system property or JNDI)
   [junit4]   2> 442827 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 442828 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 442833 INFO  (zkCallback-235-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@1f3c85af 
name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 442833 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 442833 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 442835 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 442835 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/solr.xml
   [junit4]   2> 442840 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores
   [junit4]   2> 442840 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.CoreContainer New CoreContainer 1757798446
   [junit4]   2> 442840 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/]
   [junit4]   2> 442840 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/lib
   [junit4]   2> 442840 WARN  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/lib).
   [junit4]   2> 442847 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 
90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 
20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 
2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : 
false,useRetries : false,
   [junit4]   2> 442848 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with 
params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 442849 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 442849 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 442849 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34217/solr
   [junit4]   2> 442849 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 442849 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 442854 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 442856 INFO  (zkCallback-237-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@781f7da3 
name:ZooKeeperConnection Watcher:127.0.0.1:34217 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 442857 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 442857 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 442865 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.c.ConnectionManager Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 442867 INFO  
(zkCallback-238-thread-1-processing-n:127.0.0.1:52480_f_dlr%2Fh) 
[n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@78dcafe0 
name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 442867 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.c.ConnectionManager Client is 
connected to ZooKeeper
   [junit4]   2> 442872 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.c.ZkStateReader Updating cluster 
state from ZooKeeper... 
   [junit4]   2> 443876 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.ZkController Register node as live 
in ZooKeeper:/live_nodes/127.0.0.1:52480_f_dlr%2Fh
   [junit4]   2> 443877 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:52480_f_dlr%2Fh
   [junit4]   2> 443878 INFO  
(zkCallback-231-thread-2-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ZkStateReader A live node change: 
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has 
occurred - updating... (live nodes size: 1)
   [junit4]   2> 443878 INFO  (zkCallback-233-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live 
nodes size: 1)
   [junit4]   2> 443878 INFO  
(zkCallback-231-thread-3-processing-n:127.0.0.1:58984_f_dlr%2Fh) 
[n:127.0.0.1:58984_f_dlr%2Fh    ] o.a.s.c.c.ZkStateReader A live node change: 
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has 
occurred - updating... (live nodes size: 1)
   [junit4]   2> 443878 INFO  
(zkCallback-238-thread-1-processing-n:127.0.0.1:52480_f_dlr%2Fh) 
[n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.c.ZkStateReader A live node change: 
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has 
occurred - updating... (live nodes size: 1)
   [junit4]   2> 443879 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 443880 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.LeaderElector Joined leadership 
election with path: 
/overseer_elect/election/94789660481486857-127.0.0.1:52480_f_dlr%2Fh-n_0000000001
   [junit4]   2> 443880 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.LeaderElector Watching path 
/overseer_elect/election/94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000
 to know if I could be the leader
   [junit4]   2> 443907 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.CoreContainer Security conf doesn't 
exist. Skipping setup for authorization module.
   [junit4]   2> 443907 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.CoreContainer No authentication 
plugin used.
   [junit4]   2> 443908 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.CoresLocator Looking for core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores
   [junit4]   2> 443909 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores/collection1,
 collection=collection1, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores/collection1/,
 coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 443909 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.CoresLocator Found core collection1 
in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores/collection1/
   [junit4]   2> 443909 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 443911 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 443911 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:52480_f_dlr%2Fh    ] o.a.s.s.SolrDispatchFilter 
SolrDispatchFilter.init() done
   [junit4]   2> 443913 INFO  
(coreLoadExecutor-1016-thread-1-processing-n:127.0.0.1:52480_f_dlr%2Fh) 
[n:127.0.0.1:52480_f_dlr%2Fh c:collection1   x:collection1] 
o.a.s.c.ZkController publishing state=down
   [junit4]   2> 443915 INFO  
(coreLoadExecutor-1016-thread-1-processing-n:127.0.0.1:52480_f_dlr%2Fh) 
[n:127.0.0.1:52480_f_dlr%2Fh c:collection1   x:collection1] 
o.a.s.c.ZkController numShards not found on descriptor - reading it from system 
property
   [junit4]   2> 443917 INFO  
(coreLoadExecutor-1016-thread-1-processing-n:127.0.0.1:52480_f_dlr%2Fh) 
[n:127.0.0.1:52480_f_dlr%2Fh c:collection1   x:collection1] 
o.a.s.c.ZkController Registering watch for collection collection1
   [junit4]   2> 443917 INFO  
(coreLoadExecutor-1016-thread-1-processing-n:127.0.0.1:52480_f_dlr%2Fh) 
[n:127.0.0.1:52480_f_dlr%2Fh c:collection1   x:collection1] 
o.a.s.c.c.ZkStateReader addZkWatch collection1
   [junit4]   2> 443917 INFO  (OverseerStateUpdate-94789

[...truncated too long message...]

   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:241)
   [junit4]   2>        at 
org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:85)
   [junit4]   2>        at 
org.apache.solr.cloud.ShardLeaderElectionContextBase.cancelElection(ElectionContext.java:166)
   [junit4]   2>        at 
org.apache.solr.cloud.ShardLeaderElectionContext.cancelElection(ElectionContext.java:267)
   [junit4]   2>        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:449)
   [junit4]   2>        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197)
   [junit4]   2>        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
   [junit4]   2>        at 
org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:58)
   [junit4]   2>        at 
org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:388)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:269)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 813480 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.e.j.s.ServerConnector Stopped 
ServerConnector@7ebf0b68{HTTP/1.1}{127.0.0.1:53914}
   [junit4]   2> 813481 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [    ] o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@698d4fac{/f_dlr/h,null,UNAVAILABLE}
   [junit4]   2> 813482 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A])
 [n:127.0.0.1:53914_f_dlr%2Fh c:collection1 s:shard1 r:core_node2 
x:collection1] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34217 34217
   [junit4]   2> 813683 INFO  (Thread-9093) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:34217 34217
   [junit4]   2> 813685 WARN  (Thread-9093) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        7       /solr/aliases.json
   [junit4]   2>        6       /solr/security.json
   [junit4]   2>        5       /solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        11      /solr/clusterstate.json
   [junit4]   2>        4       /solr/collections/collection1/state.json
   [junit4]   2>        4       /solr/collections/all_in_lir/state.json
   [junit4]   2>        3       
/solr/overseer_elect/election/94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        11      /solr/live_nodes
   [junit4]   2>        8       /solr/collections
   [junit4]   2>        3       /solr/overseer/queue
   [junit4]   2>        3       /solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=LeaderInitiatedRecoveryOnShardRestartTest 
-Dtests.method=testRestartWithAllInLIR -Dtests.seed=23A6EC34FC14F77A 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt 
-Dtests.locale=es_UY -Dtests.timezone=Indian/Cocos -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    373s J0 | 
LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes 
recoverying - waited for 330 seconds
   [junit4]    >        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:172)
   [junit4]    >        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:133)
   [junit4]    >        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:128)
   [junit4]    >        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:830)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR(LeaderInitiatedRecoveryOnShardRestartTest.java:81)
   [junit4]    >        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
   [junit4]    >        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)Throwable #2: 
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=10668, name=coreZkRegister-1043-thread-2, 
state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]
   [junit4]    > Caused by: java.lang.AssertionError
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([23A6EC34FC14F77A]:0)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.updateLeaderInitiatedRecoveryState(ZkController.java:2126)
   [junit4]    >        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:433)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1113)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.register(ZkController.java:926)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.register(ZkController.java:881)
   [junit4]    >        at 
org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)
   [junit4]    >        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]    >        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    >        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 813692 INFO  
(SUITE-LeaderInitiatedRecoveryOnShardRestartTest-seed#[23A6EC34FC14F77A]-worker)
 [n:127.0.0.1:53914_f_dlr%2Fh c:collection1 s:shard1 r:core_node2 
x:collection1] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001
   [junit4]   2> Nov 01, 2015 11:03:27 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene54): {}, 
docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, 
locale=es_UY, timezone=Indian/Cocos
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 
1.8.0_45 (64-bit)/cpus=4,threads=1,free=141532096,total=443023360
   [junit4]   2> NOTE: All tests run in this JVM: [TestRealTimeGet, 
RankQueryTest, TestDistributedMissingSort, TestArbitraryIndexDir, 
ExitableDirectoryReaderTest, LeaderInitiatedRecoveryOnCommitTest, 
SolrRequestParserTest, TimeZoneUtilsTest, HdfsBasicDistributedZk2Test, 
TestRemoteStreaming, DocumentAnalysisRequestHandlerTest, CacheHeaderTest, 
TestSolrConfigHandler, TestRandomDVFaceting, PolyFieldTest, 
ParsingFieldUpdateProcessorsTest, DistributedMLTComponentTest, TestConfig, 
TestAtomicUpdateErrorCases, RequiredFieldsTest, SolrCoreTest, 
PKIAuthenticationIntegrationTest, JsonLoaderTest, ResourceLoaderTest, 
TestDocBasedVersionConstraints, OverseerRolesTest, TestCloudManagedSchema, 
TestMissingGroups, LeaderInitiatedRecoveryOnShardRestartTest]
   [junit4] Completed [135/553] on J0 in 373.69s, 1 test, 1 error <<< FAILURES!

[...truncated 1359 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:782:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:719:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1432:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:987:
 There were test failures: 553 suites (8 ignored), 2176 tests, 1 error, 31 
ignored (18 assumptions) [seed: 23A6EC34FC14F77A]

Total time: 184 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to