Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/839/
1 tests failed. FAILED: org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR Error Message: Captured an uncaught exception in thread: Thread[id=10668, name=coreZkRegister-1043-thread-2, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10668, name=coreZkRegister-1043-thread-2, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest] Caused by: java.lang.AssertionError at __randomizedtesting.SeedInfo.seed([23A6EC34FC14F77A]:0) at org.apache.solr.cloud.ZkController.updateLeaderInitiatedRecoveryState(ZkController.java:2126) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:433) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1113) at org.apache.solr.cloud.ZkController.register(ZkController.java:926) at org.apache.solr.cloud.ZkController.register(ZkController.java:881) at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 10141 lines...] [junit4] Suite: org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest [junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/init-core-data-001 [junit4] 2> 440301 INFO (SUITE-LeaderInitiatedRecoveryOnShardRestartTest-seed#[23A6EC34FC14F77A]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /f_dlr/h [junit4] 2> 440305 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER [junit4] 2> 440309 INFO (Thread-9093) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 440309 INFO (Thread-9093) [ ] o.a.s.c.ZkTestServer Starting server [junit4] 2> 440409 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.ZkTestServer start zk server on port:34217 [junit4] 2> 440409 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 440412 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 440420 INFO (zkCallback-226-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7c316a08 name:ZooKeeperConnection Watcher:127.0.0.1:34217 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 440420 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 440421 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 440421 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr [junit4] 2> 440424 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 440432 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 440442 INFO (zkCallback-227-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@189070cc name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 440442 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 440443 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 440443 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1 [junit4] 2> 440444 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards [junit4] 2> 440446 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection [junit4] 2> 440447 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards [junit4] 2> 440448 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml [junit4] 2> 440448 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml [junit4] 2> 440450 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml [junit4] 2> 440450 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml [junit4] 2> 440451 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 440452 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 440453 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt [junit4] 2> 440453 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt [junit4] 2> 440454 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt [junit4] 2> 440454 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt [junit4] 2> 440455 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml [junit4] 2> 440455 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml [junit4] 2> 440457 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml [junit4] 2> 440457 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml [junit4] 2> 440458 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json [junit4] 2> 440458 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json [junit4] 2> 440459 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt [junit4] 2> 440460 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt [junit4] 2> 440464 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt [junit4] 2> 440464 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt [junit4] 2> 440466 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt [junit4] 2> 440466 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt [junit4] 2> 440802 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1 [junit4] 2> 440803 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.e.j.s.Server jetty-9.2.13.v20150730 [junit4] 2> 440805 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6111d254{/f_dlr/h,null,AVAILABLE} [junit4] 2> 440806 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.e.j.s.ServerConnector Started ServerConnector@60c4ff2b{HTTP/1.1}{127.0.0.1:58984} [junit4] 2> 440806 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.e.j.s.Server Started @443587ms [junit4] 2> 440806 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/tempDir-001/control/data, hostContext=/f_dlr/h, hostPort=58984, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores} [junit4] 2> 440807 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a [junit4] 2> 440807 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/' [junit4] 2> 440807 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx) [junit4] 2> 440807 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI) [junit4] 2> 440822 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 440824 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 440827 INFO (zkCallback-228-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3cf43ab3 name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 440827 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 440828 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 440830 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 440830 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/solr.xml [junit4] 2> 440835 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores [junit4] 2> 440835 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.CoreContainer New CoreContainer 1220789211 [junit4] 2> 440835 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/] [junit4] 2> 440835 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/lib [junit4] 2> 440835 WARN (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/lib). [junit4] 2> 440844 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false, [junit4] 2> 440845 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 440846 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 440846 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 440846 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34217/solr [junit4] 2> 440846 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.ZkController zkHost includes chroot [junit4] 2> 440846 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 440847 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 440855 INFO (zkCallback-230-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@20510216 name:ZooKeeperConnection Watcher:127.0.0.1:34217 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 440855 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 440855 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 440859 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 440881 INFO (zkCallback-231-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@12de497d name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 440881 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 440882 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue [junit4] 2> 440889 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work [junit4] 2> 440891 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running [junit4] 2> 440892 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed [junit4] 2> 440894 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure [junit4] 2> 440895 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /live_nodes [junit4] 2> 440901 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /aliases.json [junit4] 2> 440905 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json [junit4] 2> 440906 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /security.json [junit4] 2> 440907 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58984_f_dlr%2Fh [junit4] 2> 440907 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:58984_f_dlr%2Fh [junit4] 2> 440909 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect [junit4] 2> 440910 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election [junit4] 2> 440912 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.Overseer Overseer (id=null) closing [junit4] 2> 440913 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000 [junit4] 2> 440913 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58984_f_dlr%2Fh [junit4] 2> 440913 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader [junit4] 2> 440914 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.Overseer Overseer (id=94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) starting [junit4] 2> 440921 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work [junit4] 2> 440929 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000 [junit4] 2> 440938 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.Overseer Starting to work on the main queue [junit4] 2> 440938 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... [junit4] 2> 440948 INFO (OverseerCollectionConfigSetProcessor-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations [junit4] 2> 440957 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... [junit4] 2> 440970 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module. [junit4] 2> 440970 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.CoreContainer No authentication plugin used. [junit4] 2> 440971 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores [junit4] 2> 440971 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.SolrCore Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1, collection=control_collection, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=} [junit4] 2> 440972 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/ [junit4] 2> 440972 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.CoresLocator Found 1 core definitions [junit4] 2> 440977 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.ZkController publishing state=down [junit4] 2> 440978 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property [junit4] 2> 440978 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0 [junit4] 2> 440978 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done [junit4] 2> 440979 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:58984/f_dlr/h", [junit4] 2> "node_name":"127.0.0.1:58984_f_dlr%2Fh", [junit4] 2> "numShards":"1", [junit4] 2> "state":"down", [junit4] 2> "shard":null, [junit4] 2> "collection":"control_collection", [junit4] 2> "operation":"state"} current state version: 0 [junit4] 2> 440980 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.ZkController look for our core node name [junit4] 2> 440980 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={ [junit4] 2> "core":"collection1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:58984/f_dlr/h", [junit4] 2> "node_name":"127.0.0.1:58984_f_dlr%2Fh", [junit4] 2> "numShards":"1", [junit4] 2> "state":"down", [junit4] 2> "shard":null, [junit4] 2> "collection":"control_collection", [junit4] 2> "operation":"state"} [junit4] 2> 440980 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection [junit4] 2> 440981 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1 [junit4] 2> 440982 INFO (zkCallback-231-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 440996 INFO (zkCallback-231-thread-2-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 441980 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1 [junit4] 2> 441980 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection [junit4] 2> 441981 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.ZkController Collection zkNode exists [junit4] 2> 441981 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection [junit4] 2> 441982 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper [junit4] 2> 441982 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/' [junit4] 2> 441982 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx) [junit4] 2> 441982 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI) [junit4] 2> 441998 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 [junit4] 2> 442002 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1 [junit4] 2> 442006 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0 [junit4] 2> 442014 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.Config Loaded SolrConfig: solrconfig.xml [junit4] 2> 442015 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml [junit4] 2> 442020 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test [junit4] 2> 442123 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440. [junit4] 2> 442129 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.IndexSchema default search field in schema is text [junit4] 2> 442130 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.IndexSchema unique key field: id [junit4] 2> 442140 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml [junit4] 2> 442143 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml [junit4] 2> 442147 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json [junit4] 2> 442148 WARN (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE [junit4] 2> 442148 WARN (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING [junit4] 2> 442148 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json [junit4] 2> 442149 WARN (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE [junit4] 2> 442149 WARN (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING [junit4] 2> 442150 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0 [junit4] 2> 442150 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection [junit4] 2> 442150 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory [junit4] 2> 442150 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/], dataDir=[null] [junit4] 2> 442150 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@392b887 [junit4] 2> 442157 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/data [junit4] 2> 442157 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/data/index/ [junit4] 2> 442157 WARN (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index... [junit4] 2> 442157 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/data/index [junit4] 2> 442158 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@43e442ff [junit4] 2> 442164 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1 [junit4] 2> commit{dir=RawDirectoryWrapper(SimpleFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/index-SimpleFSDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@74948fa1),segFN=segments_1,generation=1} [junit4] 2> 442165 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1 [junit4] 2> 442165 INFO (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/data/ [junit4] 2> 442165 WARN (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories. [junit4] 2> 442169 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib" [junit4] 2> 442169 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe" [junit4] 2> 442169 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe" [junit4] 2> 442169 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig" [junit4] 2> 442169 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig" [junit4] 2> 442169 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit" [junit4] 2> 442170 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 442170 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 442170 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default [junit4] 2> 442172 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60 [junit4] 2> 442172 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60 [junit4] 2> 442173 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60 [junit4] 2> 442174 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60 [junit4] 2> 442177 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file [junit4] 2> 442178 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache [junit4] 2> 442181 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 442181 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 442182 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled [junit4] 2> 442182 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled [junit4] 2> 442182 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=3, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.2216796875, noCFSRatio=1.0] [junit4] 2> 442183 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1 [junit4] 2> commit{dir=RawDirectoryWrapper(SimpleFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/index-SimpleFSDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@74948fa1),segFN=segments_1,generation=1} [junit4] 2> 442183 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1 [junit4] 2> 442183 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@255ab702[collection1] main [junit4] 2> 442184 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection [junit4] 2> 442185 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper [junit4] 2> 442185 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1 [junit4] 2> 442186 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 442186 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {} [junit4] 2> 442186 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 442186 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json [junit4] 2> 442186 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 442186 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources [junit4] 2> 442187 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 442187 INFO (searcherExecutor-1006-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@255ab702[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 442187 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets [junit4] 2> 442187 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index [junit4] 2> 442187 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index [junit4] 2> 442187 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1516634569662529536 [junit4] 2> 442190 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with highest version 1516634569662529536 [junit4] 2> 442190 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1 [junit4] 2> 442190 INFO (coreLoadExecutor-1005-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1 [junit4] 2> 442191 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:58984/f_dlr/h collection:control_collection shard:shard1 [junit4] 2> 442191 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election [junit4] 2> 442195 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/94789660481486852-core_node1-n_0000000000 [junit4] 2> 442195 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1 [junit4] 2> 442197 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. [junit4] 2> 442197 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync [junit4] 2> 442197 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:58984/f_dlr/h/collection1/ [junit4] 2> 442197 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me [junit4] 2> 442197 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:58984/f_dlr/h/collection1/ has no replicas [junit4] 2> 442197 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:58984/f_dlr/h/collection1/ shard1 [junit4] 2> 442197 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection"} current state version: 1 [junit4] 2> 442197 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1 [junit4] 2> 442200 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node [junit4] 2> 442202 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "base_url":"http://127.0.0.1:58984/f_dlr/h", [junit4] 2> "core":"collection1", [junit4] 2> "state":"active"} current state version: 1 [junit4] 2> 442305 INFO (zkCallback-231-thread-2-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 442305 INFO (zkCallback-231-thread-3-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 442354 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:58984/f_dlr/h/collection1/ and leader is http://127.0.0.1:58984/f_dlr/h/collection1/ [junit4] 2> 442354 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58984/f_dlr/h [junit4] 2> 442354 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary [junit4] 2> 442354 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active [junit4] 2> 442354 INFO (coreZkRegister-999-thread-1-processing-n:127.0.0.1:58984_f_dlr%2Fh x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:58984_f_dlr%2Fh c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property [junit4] 2> 442356 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:58984/f_dlr/h", [junit4] 2> "node_name":"127.0.0.1:58984_f_dlr%2Fh", [junit4] 2> "numShards":"1", [junit4] 2> "state":"active", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "operation":"state"} current state version: 2 [junit4] 2> 442357 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={ [junit4] 2> "core":"collection1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:58984/f_dlr/h", [junit4] 2> "node_name":"127.0.0.1:58984_f_dlr%2Fh", [junit4] 2> "numShards":"1", [junit4] 2> "state":"active", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "operation":"state"} [junit4] 2> 442461 INFO (zkCallback-231-thread-3-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 442461 INFO (zkCallback-231-thread-2-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 442479 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 442480 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 442485 INFO (zkCallback-233-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@89c49b9 name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 442485 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 442485 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 442485 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... [junit4] 2> 442487 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false [junit4] 2> 442487 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2 [junit4] 2> 442487 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 442488 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 442489 INFO (zkCallback-234-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3cfc14c9 name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 442489 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 442490 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 442493 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"create", [junit4] 2> "name":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "stateFormat":"2"} current state version: 3 [junit4] 2> 442493 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1 [junit4] 2> 442596 INFO (OverseerStateUpdate-94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json [junit4] 2> 442808 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores/collection1 [junit4] 2> 442809 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001 [junit4] 2> 442810 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.e.j.s.Server jetty-9.2.13.v20150730 [junit4] 2> 442811 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3bbc54ef{/f_dlr/h,null,AVAILABLE} [junit4] 2> 442812 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.e.j.s.ServerConnector Started ServerConnector@616216a3{HTTP/1.1}{127.0.0.1:52480} [junit4] 2> 442812 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.e.j.s.Server Started @445593ms [junit4] 2> 442812 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/f_dlr/h, hostPort=52480, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores} [junit4] 2> 442812 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a [junit4] 2> 442812 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/' [junit4] 2> 442813 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx) [junit4] 2> 442813 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI) [junit4] 2> 442827 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 442828 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 442833 INFO (zkCallback-235-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1f3c85af name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 442833 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 442833 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 442835 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 442835 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/solr.xml [junit4] 2> 442840 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores [junit4] 2> 442840 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.CoreContainer New CoreContainer 1757798446 [junit4] 2> 442840 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/] [junit4] 2> 442840 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/lib [junit4] 2> 442840 WARN (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/lib). [junit4] 2> 442847 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false, [junit4] 2> 442848 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 442849 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 442849 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 442849 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34217/solr [junit4] 2> 442849 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.ZkController zkHost includes chroot [junit4] 2> 442849 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 442854 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 442856 INFO (zkCallback-237-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@781f7da3 name:ZooKeeperConnection Watcher:127.0.0.1:34217 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 442857 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 442857 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 442865 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 442867 INFO (zkCallback-238-thread-1-processing-n:127.0.0.1:52480_f_dlr%2Fh) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@78dcafe0 name:ZooKeeperConnection Watcher:127.0.0.1:34217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 442867 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 442872 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... [junit4] 2> 443876 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52480_f_dlr%2Fh [junit4] 2> 443877 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:52480_f_dlr%2Fh [junit4] 2> 443878 INFO (zkCallback-231-thread-2-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1) [junit4] 2> 443878 INFO (zkCallback-233-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1) [junit4] 2> 443878 INFO (zkCallback-231-thread-3-processing-n:127.0.0.1:58984_f_dlr%2Fh) [n:127.0.0.1:58984_f_dlr%2Fh ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1) [junit4] 2> 443878 INFO (zkCallback-238-thread-1-processing-n:127.0.0.1:52480_f_dlr%2Fh) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1) [junit4] 2> 443879 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.Overseer Overseer (id=null) closing [junit4] 2> 443880 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94789660481486857-127.0.0.1:52480_f_dlr%2Fh-n_0000000001 [junit4] 2> 443880 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000 to know if I could be the leader [junit4] 2> 443907 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module. [junit4] 2> 443907 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.CoreContainer No authentication plugin used. [junit4] 2> 443908 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores [junit4] 2> 443909 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.SolrCore Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores/collection1, collection=collection1, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=} [junit4] 2> 443909 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001/shard-1-001/cores/collection1/ [junit4] 2> 443909 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.c.CoresLocator Found 1 core definitions [junit4] 2> 443911 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0 [junit4] 2> 443911 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:52480_f_dlr%2Fh ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done [junit4] 2> 443913 INFO (coreLoadExecutor-1016-thread-1-processing-n:127.0.0.1:52480_f_dlr%2Fh) [n:127.0.0.1:52480_f_dlr%2Fh c:collection1 x:collection1] o.a.s.c.ZkController publishing state=down [junit4] 2> 443915 INFO (coreLoadExecutor-1016-thread-1-processing-n:127.0.0.1:52480_f_dlr%2Fh) [n:127.0.0.1:52480_f_dlr%2Fh c:collection1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property [junit4] 2> 443917 INFO (coreLoadExecutor-1016-thread-1-processing-n:127.0.0.1:52480_f_dlr%2Fh) [n:127.0.0.1:52480_f_dlr%2Fh c:collection1 x:collection1] o.a.s.c.ZkController Registering watch for collection collection1 [junit4] 2> 443917 INFO (coreLoadExecutor-1016-thread-1-processing-n:127.0.0.1:52480_f_dlr%2Fh) [n:127.0.0.1:52480_f_dlr%2Fh c:collection1 x:collection1] o.a.s.c.c.ZkStateReader addZkWatch collection1 [junit4] 2> 443917 INFO (OverseerStateUpdate-94789 [...truncated too long message...] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:241) [junit4] 2> at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:85) [junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.cancelElection(ElectionContext.java:166) [junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.cancelElection(ElectionContext.java:267) [junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:449) [junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) [junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157) [junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:58) [junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:388) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:269) [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> 813480 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.e.j.s.ServerConnector Stopped ServerConnector@7ebf0b68{HTTP/1.1}{127.0.0.1:53914} [junit4] 2> 813481 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@698d4fac{/f_dlr/h,null,UNAVAILABLE} [junit4] 2> 813482 INFO (TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[23A6EC34FC14F77A]) [n:127.0.0.1:53914_f_dlr%2Fh c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34217 34217 [junit4] 2> 813683 INFO (Thread-9093) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34217 34217 [junit4] 2> 813685 WARN (Thread-9093) [ ] o.a.s.c.ZkTestServer Watch limit violations: [junit4] 2> Maximum concurrent create/delete watches above limit: [junit4] 2> [junit4] 2> 7 /solr/aliases.json [junit4] 2> 6 /solr/security.json [junit4] 2> 5 /solr/configs/conf1 [junit4] 2> [junit4] 2> Maximum concurrent data watches above limit: [junit4] 2> [junit4] 2> 11 /solr/clusterstate.json [junit4] 2> 4 /solr/collections/collection1/state.json [junit4] 2> 4 /solr/collections/all_in_lir/state.json [junit4] 2> 3 /solr/overseer_elect/election/94789660481486852-127.0.0.1:58984_f_dlr%2Fh-n_0000000000 [junit4] 2> [junit4] 2> Maximum concurrent children watches above limit: [junit4] 2> [junit4] 2> 11 /solr/live_nodes [junit4] 2> 8 /solr/collections [junit4] 2> 3 /solr/overseer/queue [junit4] 2> 3 /solr/overseer/collection-queue-work [junit4] 2> [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=LeaderInitiatedRecoveryOnShardRestartTest -Dtests.method=testRestartWithAllInLIR -Dtests.seed=23A6EC34FC14F77A -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_UY -Dtests.timezone=Indian/Cocos -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1 [junit4] ERROR 373s J0 | LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR <<< [junit4] > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds [junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:172) [junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:133) [junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:128) [junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:830) [junit4] > at org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR(LeaderInitiatedRecoveryOnShardRestartTest.java:81) [junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963) [junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938) [junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10668, name=coreZkRegister-1043-thread-2, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest] [junit4] > Caused by: java.lang.AssertionError [junit4] > at __randomizedtesting.SeedInfo.seed([23A6EC34FC14F77A]:0) [junit4] > at org.apache.solr.cloud.ZkController.updateLeaderInitiatedRecoveryState(ZkController.java:2126) [junit4] > at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:433) [junit4] > at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) [junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157) [junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346) [junit4] > at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1113) [junit4] > at org.apache.solr.cloud.ZkController.register(ZkController.java:926) [junit4] > at org.apache.solr.cloud.ZkController.register(ZkController.java:881) [junit4] > at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183) [junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231) [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] 2> 813692 INFO (SUITE-LeaderInitiatedRecoveryOnShardRestartTest-seed#[23A6EC34FC14F77A]-worker) [n:127.0.0.1:53914_f_dlr%2Fh c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.SolrTestCaseJ4 ###deleteCore [junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_23A6EC34FC14F77A-001 [junit4] 2> Nov 01, 2015 11:03:27 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks [junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s). [junit4] 2> NOTE: test params are: codec=Asserting(Lucene54): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=es_UY, timezone=Indian/Cocos [junit4] 2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_45 (64-bit)/cpus=4,threads=1,free=141532096,total=443023360 [junit4] 2> NOTE: All tests run in this JVM: [TestRealTimeGet, RankQueryTest, TestDistributedMissingSort, TestArbitraryIndexDir, ExitableDirectoryReaderTest, LeaderInitiatedRecoveryOnCommitTest, SolrRequestParserTest, TimeZoneUtilsTest, HdfsBasicDistributedZk2Test, TestRemoteStreaming, DocumentAnalysisRequestHandlerTest, CacheHeaderTest, TestSolrConfigHandler, TestRandomDVFaceting, PolyFieldTest, ParsingFieldUpdateProcessorsTest, DistributedMLTComponentTest, TestConfig, TestAtomicUpdateErrorCases, RequiredFieldsTest, SolrCoreTest, PKIAuthenticationIntegrationTest, JsonLoaderTest, ResourceLoaderTest, TestDocBasedVersionConstraints, OverseerRolesTest, TestCloudManagedSchema, TestMissingGroups, LeaderInitiatedRecoveryOnShardRestartTest] [junit4] Completed [135/553] on J0 in 373.69s, 1 test, 1 error <<< FAILURES! [...truncated 1359 lines...] BUILD FAILED /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:782: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:719: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1432: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:987: There were test failures: 553 suites (8 ignored), 2176 tests, 1 error, 31 ignored (18 assumptions) [seed: 23A6EC34FC14F77A] Total time: 184 minutes 45 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts No prior successful build to compare, so performing full copy of artifacts Recording test results Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org