Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/842/

1 tests failed.
FAILED:  
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR

Error Message:
Captured an uncaught exception in thread: Thread[id=37419, 
name=coreZkRegister-7261-thread-2, state=RUNNABLE, 
group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=37419, name=coreZkRegister-7261-thread-2, 
state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]
Caused by: java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([28EDAB8B0A266015]:0)
        at 
org.apache.solr.cloud.ZkController.updateLeaderInitiatedRecoveryState(ZkController.java:2126)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:433)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1113)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:926)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:881)
        at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11042 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/init-core-data-001
   [junit4]   2> 2638758 INFO  
(SUITE-LeaderInitiatedRecoveryOnShardRestartTest-seed#[28EDAB8B0A266015]-worker)
 [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system 
property: /
   [junit4]   2> 2638760 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2638761 INFO  (Thread-27187) [    ] o.a.s.c.ZkTestServer 
client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2638761 INFO  (Thread-27187) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 2638861 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.ZkTestServer start zk server on port:46279
   [junit4]   2> 2638861 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2638862 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2638865 INFO  (zkCallback-2032-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@1ed99d8a 
name:ZooKeeperConnection Watcher:127.0.0.1:46279 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2638865 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2638866 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2638866 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 2638868 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2638869 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2638869 INFO  (zkCallback-2033-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@6cbb5154 
name:ZooKeeperConnection Watcher:127.0.0.1:46279/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2638869 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2638869 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2638869 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 2638871 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 2638872 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 2638873 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 2638874 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 2638874 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2638876 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 2638876 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 2638878 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2638878 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2638879 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 2638879 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2638880 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 2638880 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2638882 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 2638882 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 2638883 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2638883 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2638884 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2638885 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2638886 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2638886 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2638887 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2638887 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2638888 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 2638889 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2639226 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores/collection1
   [junit4]   2> 2639227 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 2639229 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@233dba17{/,null,AVAILABLE}
   [junit4]   2> 2639230 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.e.j.s.ServerConnector Started 
ServerConnector@cc5a419{HTTP/1.1}{127.0.0.1:48688}
   [junit4]   2> 2639230 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.e.j.s.Server Started @2642378ms
   [junit4]   2> 2639230 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/tempDir-001/control/data,
 hostContext=/, hostPort=48688, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores}
   [junit4]   2> 2639230 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 2639230 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/'
   [junit4]   2> 2639231 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr 
(NoInitialContextEx)
   [junit4]   2> 2639231 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not 
find system property or JNDI)
   [junit4]   2> 2639246 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2639247 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2639248 INFO  (zkCallback-2034-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@61fbe622 
name:ZooKeeperConnection Watcher:127.0.0.1:46279/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2639248 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2639249 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2639249 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 2639249 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/solr.xml
   [junit4]   2> 2639254 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores
   [junit4]   2> 2639254 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.CoreContainer New CoreContainer 601359846
   [junit4]   2> 2639254 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/]
   [junit4]   2> 2639255 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/lib
   [junit4]   2> 2639255 WARN  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/lib).
   [junit4]   2> 2639262 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 
90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 
20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 
2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : 
false,useRetries : false,
   [junit4]   2> 2639263 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with 
params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2639264 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2639264 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2639264 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46279/solr
   [junit4]   2> 2639264 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 2639264 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2639264 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2639265 INFO  (zkCallback-2036-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@1831b96b 
name:ZooKeeperConnection Watcher:127.0.0.1:46279 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2639265 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2639266 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2639267 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.ConnectionManager Waiting for client to 
connect to ZooKeeper
   [junit4]   2> 2639267 INFO  
(zkCallback-2037-thread-1-processing-n:127.0.0.1:48688_) [n:127.0.0.1:48688_    
] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@1cf94f83 
name:ZooKeeperConnection Watcher:127.0.0.1:46279/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2639267 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.ConnectionManager Client is connected to 
ZooKeeper
   [junit4]   2> 2639268 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 2639270 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-queue-work
   [junit4]   2> 2639272 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-running
   [junit4]   2> 2639273 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-completed
   [junit4]   2> 2639275 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-failure
   [junit4]   2> 2639277 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 2639278 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 2639279 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 2639280 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 2639280 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:48688_
   [junit4]   2> 2639281 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:48688_
   [junit4]   2> 2639285 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 2639287 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/election
   [junit4]   2> 2639288 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2639288 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.LeaderElector Joined leadership election with 
path: /overseer_elect/election/94806770325520388-127.0.0.1:48688_-n_0000000000
   [junit4]   2> 2639288 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.OverseerElectionContext I am going to be the 
leader 127.0.0.1:48688_
   [junit4]   2> 2639289 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/leader
   [junit4]   2> 2639290 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.Overseer Overseer 
(id=94806770325520388-127.0.0.1:48688_-n_0000000000) starting
   [junit4]   2> 2639291 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 2639295 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting 
OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=30000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2639296 INFO  
(OverseerCollectionConfigSetProcessor-94806770325520388-127.0.0.1:48688_-n_0000000000)
 [n:127.0.0.1:48688_    ] o.a.s.c.OverseerTaskProcessor Process current queue 
of overseer operations
   [junit4]   2> 2639296 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 2639297 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.c.ZkStateReader Updating cluster state from 
ZooKeeper... 
   [junit4]   2> 2639301 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.c.ZkStateReader Updating cluster state from 
ZooKeeper... 
   [junit4]   2> 2639313 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.CoreContainer Security conf doesn't exist. 
Skipping setup for authorization module.
   [junit4]   2> 2639313 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 2639314 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.CoresLocator Looking for core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores
   [junit4]   2> 2639314 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores/collection1,
 collection=control_collection, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores/collection1/,
 coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 2639314 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.CoresLocator Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores/collection1/
   [junit4]   2> 2639314 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 2639316 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 2639316 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:48688_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() 
done
   [junit4]   2> 2639316 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.c.ZkController 
publishing state=down
   [junit4]   2> 2639316 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.c.ZkController 
numShards not found on descriptor - reading it from system property
   [junit4]   2> 2639318 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:48688";,
   [junit4]   2>   "node_name":"127.0.0.1:48688_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2639318 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.c.ZkController 
look for our core node name
   [junit4]   2> 2639318 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:48688";,
   [junit4]   2>   "node_name":"127.0.0.1:48688_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 2639318 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.o.ClusterStateMutator building a new cName: 
control_collection
   [junit4]   2> 2639319 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard 
shard=shard1
   [junit4]   2> 2639320 INFO  
(zkCallback-2037-thread-1-processing-n:127.0.0.1:48688_) [n:127.0.0.1:48688_    
] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred 
- updating... (live nodes size: 1)
   [junit4]   2> 2639320 INFO  
(zkCallback-2037-thread-2-processing-n:127.0.0.1:48688_) [n:127.0.0.1:48688_    
] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred 
- updating... (live nodes size: 1)
   [junit4]   2> 2640318 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.c.ZkController 
waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2640318 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.c.ZkController 
Check for collection zkNode:control_collection
   [junit4]   2> 2640319 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.c.ZkController 
Collection zkNode exists
   [junit4]   2> 2640319 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 2640320 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 
specified config exists in ZooKeeper
   [junit4]   2> 2640320 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores/collection1/'
   [junit4]   2> 2640320 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2640320 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 2640332 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.c.Config loaded 
config solrconfig.xml with version 0 
   [junit4]   2> 2640337 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
current version of requestparams : -1
   [junit4]   2> 2640340 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2640357 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.c.Config Loaded 
SolrConfig: solrconfig.xml
   [junit4]   2> 2640358 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 2640363 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 2640453 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Initialized with 
rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2640458 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
default search field in schema is text
   [junit4]   2> 2640459 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
unique key field: id
   [junit4]   2> 2640468 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 2640470 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 2640471 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 2640472 WARN  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 2640472 WARN  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 2640472 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 2640473 WARN  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 2640473 WARN  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 2640474 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] 
o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default 
properties: Can't find resource 'configsetprops.json' in classpath or 
'/configs/conf1', 
cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 2640474 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection   x:collection1] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1' using configuration from collection 
control_collection
   [junit4]   2> 2640474 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2640475 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores/collection1/],
 dataDir=[null]
   [junit4]   2> 2640475 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ba862f0
   [junit4]   2> 2640475 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.CachingDirectoryFactory return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores/collection1/data
   [junit4]   2> 2640475 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore New index directory detected: old=null 
new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores/collection1/data/index/
   [junit4]   2> 2640475 WARN  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore [collection1] Solr index directory 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores/collection1/data/index'
 doesn't exist. Creating new index...
   [junit4]   2> 2640476 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.CachingDirectoryFactory return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores/collection1/data/index
   [junit4]   2> 2640476 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=17, maxMergeAtOnceExplicit=35, maxMergedSegmentMB=81.392578125, 
floorSegmentMB=2.0615234375, forceMergeDeletesPctAllowed=29.2956960078632, 
segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.13545458923845866
   [junit4]   2> 2640476 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>        commit{dir=MockDirectoryWrapper(RAMDirectory@1591bda2 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2f2b28aa),segFN=segments_1,generation=1}
   [junit4]   2> 2640476 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 2640477 INFO  
(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
Looking for old index directories to cleanup for core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores/collection1/data/
   [junit4]   2> 2640477 WARN  
(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.DirectoryFactory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/control-001/cores/collection1/data/
 does not point to a valid data directory; skipping clean-up of old index 
directories.
   [junit4]   2> 2640479 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"nodistrib"
   [junit4]   2> 2640479 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"dedupe"
   [junit4]   2> 2640480 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2640480 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"stored_sig"
   [junit4]   2> 2640480 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2640480 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"distrib-dup-test-chain-explicit"
   [junit4]   2> 2640480 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 2640480 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 2640480 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating 
implicit default
   [junit4]   2> 2640480 WARN  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 2640482 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2640483 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2640484 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2640485 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2640487 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.RequestHandlers Registered paths: 
/admin/mbeans,standard,/dump,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 2640487 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore Using default statsCache cache: 
org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 2640488 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 2640488 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2640489 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 2640489 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 2640489 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=25, maxMergeAtOnceExplicit=28, maxMergedSegmentMB=25.2373046875, 
floorSegmentMB=0.9736328125, forceMergeDeletesPctAllowed=18.96166296556762, 
segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2640489 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>        commit{dir=MockDirectoryWrapper(RAMDirectory@1591bda2 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2f2b28aa),segFN=segments_1,generation=1}
   [junit4]   2> 2640490 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 2640490 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.s.SolrIndexSearcher Opening Searcher@4e110cbd[collection1] main
   [junit4]   2> 2640490 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 2640491 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 
specified config exists in ZooKeeper
   [junit4]   2> 2640491 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the 
RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2640491 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 2640491 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 2640491 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2640492 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage No data found for znode 
/configs/conf1/_rest_managed.json
   [junit4]   2> 2640492 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2640492 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 2640492 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2640492 INFO  
(searcherExecutor-7224-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
[collection1] Registered new searcher Searcher@4e110cbd[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2640493 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 2640493 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version 
buckets from index
   [junit4]   2> 2640493 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket 
highest value from index
   [junit4]   2> 2640493 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1516908327023411200
   [junit4]   2> 2640495 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with highest version 
1516908327023411200
   [junit4]   2> 2640495 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 2640495 INFO  
(coreLoadExecutor-7223-thread-1-processing-n:127.0.0.1:48688_) 
[n:127.0.0.1:48688_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 2640496 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
Register replica - core:collection1 address:http://127.0.0.1:48688 
collection:control_collection shard:shard1
   [junit4]   2> 2640497 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.SolrZkClient makePath: 
/collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2640502 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector 
Joined leadership election with path: 
/collections/control_collection/leader_elect/shard1/election/94806770325520388-core_node1-n_0000000000
   [junit4]   2> 2640502 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 2640504 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 2640505 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2640505 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2640505 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:48688/collection1/
   [junit4]   2> 2640505 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
Sync Success - now sync replicas to me
   [junit4]   2> 2640505 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
http://127.0.0.1:48688/collection1/ has no replicas
   [junit4]   2> 2640505 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:48688/collection1/ shard1
   [junit4]   2> 2640506 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2640508 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 2640511 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:48688";,
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 2640613 INFO  
(zkCallback-2037-thread-2-processing-n:127.0.0.1:48688_) [n:127.0.0.1:48688_    
] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred 
- updating... (live nodes size: 1)
   [junit4]   2> 2640613 INFO  
(zkCallback-2037-thread-1-processing-n:127.0.0.1:48688_) [n:127.0.0.1:48688_    
] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred 
- updating... (live nodes size: 1)
   [junit4]   2> 2640662 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
We are http://127.0.0.1:48688/collection1/ and leader is 
http://127.0.0.1:48688/collection1/
   [junit4]   2> 2640662 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48688
   [junit4]   2> 2640662 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
I am the leader, no recovery necessary
   [junit4]   2> 2640663 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
publishing state=active
   [junit4]   2> 2640663 INFO  
(coreZkRegister-7217-thread-1-processing-n:127.0.0.1:48688_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:48688_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
numShards not found on descriptor - reading it from system property
   [junit4]   2> 2640669 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:48688";,
   [junit4]   2>   "node_name":"127.0.0.1:48688_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 2640670 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:48688";,
   [junit4]   2>   "node_name":"127.0.0.1:48688_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 2640772 INFO  
(zkCallback-2037-thread-2-processing-n:127.0.0.1:48688_) [n:127.0.0.1:48688_    
] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred 
- updating... (live nodes size: 1)
   [junit4]   2> 2640772 INFO  
(zkCallback-2037-thread-1-processing-n:127.0.0.1:48688_) [n:127.0.0.1:48688_    
] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred 
- updating... (live nodes size: 1)
   [junit4]   2> 2640817 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2640818 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2640819 INFO  (zkCallback-2039-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@7d90c15b 
name:ZooKeeperConnection Watcher:127.0.0.1:46279/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2640819 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2640819 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2640819 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2640821 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause 
connection loss:false
   [junit4]   2> 2640821 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with 
stateFormat=2
   [junit4]   2> 2640821 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2640822 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2640823 INFO  (zkCallback-2040-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@204c2203 
name:ZooKeeperConnection Watcher:127.0.0.1:46279/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2640823 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2640823 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2640825 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2"} current state version: 3
   [junit4]   2> 2640825 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.o.ClusterStateMutator building a new cName: 
collection1
   [junit4]   2> 2640930 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.o.ZkStateWriter going to create_collection 
/collections/collection1/state.json
   [junit4]   2> 2641147 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/cores/collection1
   [junit4]   2> 2641148 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001
   [junit4]   2> 2641149 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 2641151 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@65089103{/,null,AVAILABLE}
   [junit4]   2> 2641151 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.e.j.s.ServerConnector Started 
ServerConnector@342effb3{HTTP/1.1}{127.0.0.1:42846}
   [junit4]   2> 2641151 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.e.j.s.Server Started @2644299ms
   [junit4]   2> 2641151 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/tempDir-001/jetty1,
 solrconfig=solrconfig.xml, hostContext=/, hostPort=42846, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/cores}
   [junit4]   2> 2641152 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 2641152 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/'
   [junit4]   2> 2641152 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr 
(NoInitialContextEx)
   [junit4]   2> 2641152 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not 
find system property or JNDI)
   [junit4]   2> 2641165 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2641165 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2641166 INFO  (zkCallback-2041-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@4e4c250e 
name:ZooKeeperConnection Watcher:127.0.0.1:46279/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2641167 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2641167 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2641168 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 2641168 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/solr.xml
   [junit4]   2> 2641172 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/cores
   [junit4]   2> 2641173 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.CoreContainer New CoreContainer 480186408
   [junit4]   2> 2641173 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/]
   [junit4]   2> 2641173 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/lib
   [junit4]   2> 2641173 WARN  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/lib).
   [junit4]   2> 2641180 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 
90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 
20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 
2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : 
false,useRetries : false,
   [junit4]   2> 2641181 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with 
params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2641181 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2641181 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2641182 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46279/solr
   [junit4]   2> 2641182 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 2641182 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2641182 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2641183 INFO  (zkCallback-2043-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@27e5d14f 
name:ZooKeeperConnection Watcher:127.0.0.1:46279 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2641183 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2641184 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2641185 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.c.ConnectionManager Waiting for client to 
connect to ZooKeeper
   [junit4]   2> 2641186 INFO  
(zkCallback-2044-thread-1-processing-n:127.0.0.1:42846_) [n:127.0.0.1:42846_    
] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@741158e4 
name:ZooKeeperConnection Watcher:127.0.0.1:46279/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2641186 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.c.ConnectionManager Client is connected to 
ZooKeeper
   [junit4]   2> 2641190 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.c.ZkStateReader Updating cluster state from 
ZooKeeper... 
   [junit4]   2> 2642197 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:42846_
   [junit4]   2> 2642197 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:42846_
   [junit4]   2> 2642198 INFO  (zkCallback-2039-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live 
nodes size: 1)
   [junit4]   2> 2642198 INFO  
(zkCallback-2037-thread-1-processing-n:127.0.0.1:48688_) [n:127.0.0.1:48688_    
] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live 
nodes size: 1)
   [junit4]   2> 2642198 INFO  
(zkCallback-2037-thread-2-processing-n:127.0.0.1:48688_) [n:127.0.0.1:48688_    
] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live 
nodes size: 1)
   [junit4]   2> 2642199 INFO  
(zkCallback-2044-thread-1-processing-n:127.0.0.1:42846_) [n:127.0.0.1:42846_    
] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live 
nodes size: 1)
   [junit4]   2> 2642199 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2642200 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.LeaderElector Joined leadership election with 
path: /overseer_elect/election/94806770325520393-127.0.0.1:42846_-n_0000000001
   [junit4]   2> 2642201 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.LeaderElector Watching path 
/overseer_elect/election/94806770325520388-127.0.0.1:48688_-n_0000000000 to 
know if I could be the leader
   [junit4]   2> 2642208 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.CoreContainer Security conf doesn't exist. 
Skipping setup for authorization module.
   [junit4]   2> 2642208 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 2642209 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.CoresLocator Looking for core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/cores
   [junit4]   2> 2642210 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/cores/collection1,
 collection=collection1, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/cores/collection1/,
 coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 2642210 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.CoresLocator Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/cores/collection1/
   [junit4]   2> 2642210 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 2642211 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 2642211 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[28EDAB8B0A266015])
 [n:127.0.0.1:42846_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() 
done
   [junit4]   2> 2642211 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.ZkController 
publishing state=down
   [junit4]   2> 2642211 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.ZkController 
numShards not found on descriptor - reading it from system property
   [junit4]   2> 2642213 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.ZkController 
Registering watch for collection collection1
   [junit4]   2> 2642213 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader 
addZkWatch collection1
   [junit4]   2> 2642213 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:42846";,
   [junit4]   2>   "node_name":"127.0.0.1:42846_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 2642213 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Add 
data for collection1 ver 0 
   [junit4]   2> 2642213 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.ZkController look 
for our core node name
   [junit4]   2> 2642213 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:42846";,
   [junit4]   2>   "node_name":"127.0.0.1:42846_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 2642214 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.o.ReplicaMutator Collection already exists 
with numShards=1
   [junit4]   2> 2642214 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard 
shard=shard1
   [junit4]   2> 2642316 INFO  
(OverseerStateUpdate-94806770325520388-127.0.0.1:48688_-n_0000000000) 
[n:127.0.0.1:48688_    ] o.a.s.c.o.ZkStateWriter going to update_collection 
/collections/collection1/state.json version: 0
   [junit4]   2> 2642316 INFO  
(zkCallback-2044-thread-1-processing-n:127.0.0.1:42846_) [n:127.0.0.1:42846_    
] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json for collection collection1 has 
occurred - updating... (live nodes size: 2)
   [junit4]   2> 2642317 INFO  
(zkCallback-2044-thread-1-processing-n:127.0.0.1:42846_) [n:127.0.0.1:42846_    
] o.a.s.c.c.ZkStateReader Updating data for collection1 from 0 to 1 
   [junit4]   2> 2643213 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.ZkController waiting 
to find shard id in clusterstate for collection1
   [junit4]   2> 2643213 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.ZkController Check 
for collection zkNode:collection1
   [junit4]   2> 2643214 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.ZkController 
Collection zkNode exists
   [junit4]   2> 2643214 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Load 
collection config from:/collections/collection1
   [junit4]   2> 2643214 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader 
path=/collections/collection1 configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 2643215 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.SolrResourceLoader 
new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001/shard-1-001/cores/collection1/'
   [junit4]   2> 2643215 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.SolrResourceLoader 
JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2643215 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collection1] o.a.s.c.SolrResourceLoader 
solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 2643234 INFO  
(coreLoadExecutor-7234-thread-1-processing-n:127.0.0.1:42846_) 
[n:127.0.0.1:42846_ c:collection1   x:collect

[...truncated too long message...]

tRestartWithAllInLIR <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes 
recoverying - waited for 330 seconds
   [junit4]    >        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:172)
   [junit4]    >        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:133)
   [junit4]    >        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:128)
   [junit4]    >        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:830)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR(LeaderInitiatedRecoveryOnShardRestartTest.java:81)
   [junit4]    >        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
   [junit4]    >        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)Throwable #2: 
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=37419, name=coreZkRegister-7261-thread-2, 
state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]
   [junit4]    > Caused by: java.lang.AssertionError
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([28EDAB8B0A266015]:0)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.updateLeaderInitiatedRecoveryState(ZkController.java:2126)
   [junit4]    >        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:433)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1113)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.register(ZkController.java:926)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.register(ZkController.java:881)
   [junit4]    >        at 
org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)
   [junit4]    >        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]    >        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    >        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 3009479 INFO  
(SUITE-LeaderInitiatedRecoveryOnShardRestartTest-seed#[28EDAB8B0A266015]-worker)
 [n:127.0.0.1:60985_ c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_28EDAB8B0A266015-001
   [junit4]   2> Nov 04, 2015 11:34:40 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: 
codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY,
 chunkSize=32730, maxDocsPerChunk=1, blockSize=5), 
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, 
chunkSize=32730, blockSize=5)), sim=ClassicSimilarity, locale=en_MT, 
timezone=America/Indiana/Vevay
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 
1.8.0_45 (64-bit)/cpus=4,threads=1,free=276098240,total=507510784
   [junit4]   2> NOTE: All tests run in this JVM: [CurrencyFieldXmlFileTest, 
TestSolrCLIRunExample, HighlighterConfigTest, TestRawTransformer, 
DistributedDebugComponentTest, BasicDistributedZk2Test, 
TestShortCircuitedRequests, FieldAnalysisRequestHandlerTest, 
AddSchemaFieldsUpdateProcessorFactoryTest, HLLSerializationTest, 
TestJsonFacets, TestCustomSort, ZkStateReaderTest, 
TestOverriddenPrefixQueryForCustomFieldType, 
DeleteLastCustomShardedReplicaTest, TestCursorMarkWithoutUniqueKey, 
CleanupOldIndexTest, DistributedQueryComponentCustomSortTest, 
TestCSVResponseWriter, SampleTest, OverriddenZkACLAndCredentialsProvidersTest, 
TestFieldTypeCollectionResource, TestSearchPerf, BJQParserTest, 
SecurityConfHandlerTest, ResponseHeaderTest, DeleteShardTest, 
DebugComponentTest, TestScoreJoinQPNoScore, TestLeaderInitiatedRecoveryThread, 
TestPostingsSolrHighlighter, ClusterStateTest, JavabinLoaderTest, FullHLLTest, 
RegexBoostProcessorTest, TestOrdValues, FullSolrCloudDistribCmdsTest, 
TestFastLRUCache, DistributedFacetPivotSmallAdvancedTest, TestCollationField, 
NoCacheHeaderTest, TestPerFieldSimilarity, TestStressReorder, 
LukeRequestHandlerTest, TestCharFilters, TestXIncludeConfig, 
SystemInfoHandlerTest, TestJmxIntegration, ZkNodePropsTest, TestReqParamsAPI, 
CdcrUpdateLogTest, OverseerCollectionConfigSetProcessorTest, AutoCommitTest, 
ConcurrentDeleteAndCreateCollectionTest, TestAuthenticationFramework, 
PluginInfoTest, TestManagedResource, TestFieldTypeResource, 
TriLevelCompositeIdRoutingTest, QueryResultKeyTest, TestElisionMultitermQuery, 
BadIndexSchemaTest, AnalyticsQueryTest, TestManagedStopFilterFactory, 
SolrCoreCheckLockOnStartupTest, ShowFileRequestHandlerTest, 
CoreAdminHandlerTest, DistributedFacetPivotSmallTest, HdfsDirectoryTest, 
TestSearcherReuse, HardAutoCommitTest, TestSort, 
CloudExitableDirectoryReaderTest, TestLRUCache, StressHdfsTest, RecoveryZkTest, 
TestIndexingPerformance, TestFieldSortValues, BinaryUpdateRequestHandlerTest, 
DirectUpdateHandlerTest, TestMiniSolrCloudClusterBase, UnloadDistributedZkTest, 
TestReversedWildcardFilterFactory, BasicAuthIntegrationTest, SuggesterTest, 
AsyncCallRequestStatusResponseTest, TestIndexSearcher, TestRecovery, 
TestTrieFacet, ChaosMonkeySafeLeaderTest, OverseerTest, 
LeaderElectionIntegrationTest, BasicZkTest, LeaderElectionTest, TestZkChroot, 
TestRandomDVFaceting, ZkCLITest, ShardRoutingCustomTest, TestFaceting, 
TestHashPartitioner, TermVectorComponentDistributedTest, ZkControllerTest, 
TestRealTimeGet, TestReload, TestStressVersions, TestRangeQuery, 
TestCoreContainer, SimpleFacetsTest, TestSolr4Spatial, SolrCoreTest, 
StatsComponentTest, SpellCheckComponentTest, TestGroupingSearch, 
QueryElevationComponentTest, TestLazyCores, SoftAutoCommitTest, 
CurrencyFieldOpenExchangeTest, SolrIndexSplitterTest, SimplePostToolTest, 
TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, 
TestExtendedDismaxParser, SpellCheckCollatorTest, SuggesterFSTTest, 
SolrRequestParserTest, TestFoldingMultitermQuery, DocValuesTest, 
SuggesterTSTTest, SuggesterWFSTTest, TestCSVLoader, 
SchemaVersionSpecificBehaviorTest, FieldMutatingUpdateProcessorTest, 
TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, 
DirectUpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, 
DocValuesMultiTest, DistanceFunctionTest, TestSolrDeletionPolicy1, 
SolrInfoMBeanTest, XsltUpdateRequestHandlerTest, CacheHeaderTest, 
IndexBasedSpellCheckerTest, TestSurroundQueryParser, DisMaxRequestHandlerTest, 
TestWriterPerf, TestQueryTypes, PrimitiveFieldTypeTest, TestOmitPositions, 
XmlUpdateRequestHandlerTest, RequestHandlersTest, DocumentBuilderTest, 
TestValueSourceCache, PathHierarchyTokenizerFactoryTest, 
MoreLikeThisHandlerTest, TestSolrQueryParser, TestArbitraryIndexDir, 
IndexSchemaRuntimeFieldTest, LoggingHandlerTest, SolrPluginUtilsTest, 
ReturnFieldsTest, UpdateRequestProcessorFactoryTest, JSONWriterTest, 
QueryParsingTest, TestAnalyzedSuggestions, JsonLoaderTest, 
TestPHPSerializedResponseWriter, CSVRequestHandlerTest, SearchHandlerTest, 
TestLFUCache, TestQuerySenderListener, UpdateParamsTest, 
AlternateDirectoryTest, TestSolrIndexConfig, TestQuerySenderNoQuery, 
CopyFieldTest, SolrIndexConfigTest, BadComponentTest, TestStressRecovery, 
SolrTestCaseJ4Test, TestCrossCoreJoin, TestDistributedMissingSort, 
TestEmbeddedSolrServerConstructors, AliasIntegrationTest, 
AsyncMigrateRouteKeyTest, CdcrVersionReplicationTest, 
CollectionsAPIAsyncDistributedZkTest, CustomCollectionTest, 
DistribCursorPagingTest, LeaderInitiatedRecoveryOnCommitTest, 
LeaderInitiatedRecoveryOnShardRestartTest]
   [junit4] Completed [412/554] on J2 in 370.99s, 1 test, 1 error <<< FAILURES!

[...truncated 476 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:782:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:719:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1432:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:987:
 There were test failures: 554 suites (8 ignored), 2182 tests, 1 error, 31 
ignored (18 assumptions) [seed: 28EDAB8B0A266015]

Total time: 190 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to