Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/892/

1 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest.test

Error Message:
Address already in use

Stack Trace:
java.net.BindException: Address already in use
        at 
__randomizedtesting.SeedInfo.seed([B49EC28476BD3CBD:3CCAFD5ED8415145]:0)
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at 
org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:326)
        at 
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
        at 
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:244)
        at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.server.Server.doStart(Server.java:384)
        at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:412)
        at org.apache.solr.cloud.RecoveryZkTest.test(RecoveryZkTest.java:91)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:965)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11452 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/init-core-data-001
   [junit4]   2> 3577534 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 3577534 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 3577614 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] 
o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 3577619 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3577621 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] o.m.log 
jetty-6.1.26
   [junit4]   2> 3577636 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] o.m.log 
Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_47408_hdfs____.edyanx/webapp
   [junit4]   2> 3577747 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] o.m.log NO JSP 
Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 3577949 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] o.m.log 
Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47408
   [junit4]   2> 3578090 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3578103 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] o.m.log 
jetty-6.1.26
   [junit4]   2> 3578142 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] o.m.log 
Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_55171_datanode____pvytua/webapp
   [junit4]   2> 3578281 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] o.m.log NO JSP 
Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 3578623 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] o.m.log 
Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:55171
   [junit4]   2> 3578772 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3578773 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] o.m.log 
jetty-6.1.26
   [junit4]   2> 3578839 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] o.m.log 
Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_42039_datanode____.1gxght/webapp
   [junit4]   2> 3578884 INFO  (IPC Server handler 3 on 52056) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-8c9de557-5e09-47af-a3c6-686be8aae2ec node DatanodeRegistration(127.0.0.1, 
datanodeUuid=e0bd79f9-fc78-4cf6-90d6-92b57484246b, infoPort=55171, 
ipcPort=51787, storageInfo=lv=-56;cid=testClusterID;nsid=1426836081;c=0), 
blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 3578884 INFO  (IPC Server handler 3 on 52056) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-135f82ae-10b6-424f-9c1e-20e1ae53c3eb node DatanodeRegistration(127.0.0.1, 
datanodeUuid=e0bd79f9-fc78-4cf6-90d6-92b57484246b, infoPort=55171, 
ipcPort=51787, storageInfo=lv=-56;cid=testClusterID;nsid=1426836081;c=0), 
blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 3579180 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] o.m.log NO JSP 
Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 3579458 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [    ] o.m.log 
Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42039
   [junit4]   2> 3579673 INFO  (IPC Server handler 5 on 52056) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-9ca899b2-5e33-4f0a-92bf-164577124e88 node DatanodeRegistration(127.0.0.1, 
datanodeUuid=91aa6a45-1d17-4562-831f-b714a2cac565, infoPort=42039, 
ipcPort=35292, storageInfo=lv=-56;cid=testClusterID;nsid=1426836081;c=0), 
blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 3579674 INFO  (IPC Server handler 5 on 52056) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-4299d711-c066-4898-b227-aefb31f022c0 node DatanodeRegistration(127.0.0.1, 
datanodeUuid=91aa6a45-1d17-4562-831f-b714a2cac565, infoPort=42039, 
ipcPort=35292, storageInfo=lv=-56;cid=testClusterID;nsid=1426836081;c=0), 
blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 3579890 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3579891 INFO  (Thread-14444) [    ] o.a.s.c.ZkTestServer 
client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3579891 INFO  (Thread-14444) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 3579991 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:59246
   [junit4]   2> 3579992 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3579993 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3579998 INFO  (zkCallback-2836-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@743e874b 
name:ZooKeeperConnection Watcher:127.0.0.1:59246 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3579998 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3579998 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3579998 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 3580001 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3580003 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3580006 INFO  (zkCallback-2837-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@52ce9478 
name:ZooKeeperConnection Watcher:127.0.0.1:59246/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3580006 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3580006 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3580006 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 3580010 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 3580015 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 3580017 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 3580034 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 3580035 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 3580038 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 3580039 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 3580041 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3580041 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3580049 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 3580049 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 3580077 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 3580078 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 3580097 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 3580097 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 3580104 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3580104 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 3580106 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3580106 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3580108 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3580108 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3580109 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3580109 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 3580111 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 3580111 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 3580927 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/control-001/cores/collection1
   [junit4]   2> 3580929 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] o.e.j.s.Server 
jetty-9.3.6.v20151106
   [junit4]   2> 3580932 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@6f5dfe2b{/,null,AVAILABLE}
   [junit4]   2> 3580932 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@52602ee0{HTTP/1.1,[http/1.1]}{127.0.0.1:47355}
   [junit4]   2> 3580932 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] o.e.j.s.Server 
Started @3584136ms
   [junit4]   2> 3580932 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=hdfs://localhost:52056/hdfs__localhost_52056__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-trunk_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001_tempDir-002_control_data,
 hostContext=/, hostPort=47355, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/control-001/cores}
   [junit4]   2> 3580933 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 3580933 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/control-001'
   [junit4]   2> 3580933 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3580933 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 3580933 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3580936 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3580938 INFO  (zkCallback-2838-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@17ea5c8d 
name:ZooKeeperConnection Watcher:127.0.0.1:59246/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3580938 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3580939 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3580940 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 3580940 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/control-001/solr.xml
   [junit4]   2> 3580951 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.CorePropertiesLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/control-001/cores
   [junit4]   2> 3580951 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.CoreContainer New CoreContainer 1305483320
   [junit4]   2> 3580952 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/control-001]
   [junit4]   2> 3580952 WARN  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.CoreContainer Couldn't add files from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/control-001/lib
 to classpath: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/control-001/lib
   [junit4]   2> 3580952 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme 
: ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 
10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 
5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 3580954 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3580956 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3580956 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3580956 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59246/solr
   [junit4]   2> 3580956 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 3580956 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3580956 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3580958 INFO  (zkCallback-2841-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@48bc42ea 
name:ZooKeeperConnection Watcher:127.0.0.1:59246 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3580958 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3580958 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3580960 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3580961 INFO  
(zkCallback-2842-thread-1-processing-n:127.0.0.1:47355_) [n:127.0.0.1:47355_    
] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@4301550b 
name:ZooKeeperConnection Watcher:127.0.0.1:59246/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3580961 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3580962 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 3580965 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 3580968 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 3580970 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 3580972 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 3580974 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 3580975 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 3580976 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 3580977 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 3580979 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:47355_
   [junit4]   2> 3580979 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:47355_
   [junit4]   2> 3580981 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 3580982 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 3580983 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3580983 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.LeaderElector Joined leadership election with path: 
/overseer_elect/election/95097466067353604-127.0.0.1:47355_-n_0000000000
   [junit4]   2> 3580984 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:47355_
   [junit4]   2> 3580984 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 3580985 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.Overseer Overseer (id=95097466067353604-127.0.0.1:47355_-n_0000000000) 
starting
   [junit4]   2> 3580990 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 3580998 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.OverseerAutoReplicaFailoverThread Starting 
OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=30000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 3580999 INFO  
(OverseerCollectionConfigSetProcessor-95097466067353604-127.0.0.1:47355_-n_0000000000)
 [n:127.0.0.1:47355_    ] o.a.s.c.OverseerTaskProcessor Process current queue 
of overseer operations
   [junit4]   2> 3581000 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 3581001 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.c.ZkStateReader Updating cluster state from 
ZooKeeper... 
   [junit4]   2> 3581006 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3581030 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for 
authorization module.
   [junit4]   2> 3581031 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 3581031 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.CorePropertiesLocator Looking for core definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/control-001/cores
   [junit4]   2> 3581032 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, 
config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, 
configSetProperties=configsetprops.json, collection=control_collection, 
coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 3581032 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.CorePropertiesLocator Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/control-001/cores/collection1
   [junit4]   2> 3581033 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 3581035 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 3581035 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:47355_    ] 
o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 3581035 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.c.ZkController 
publishing state=down
   [junit4]   2> 3581035 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.c.ZkController 
numShards not found on descriptor - reading it from system property
   [junit4]   2> 3581038 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.c.ZkController 
look for our core node name
   [junit4]   2> 3581039 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:47355";,
   [junit4]   2>   "node_name":"127.0.0.1:47355_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 3581039 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:47355";,
   [junit4]   2>   "node_name":"127.0.0.1:47355_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 3581040 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.o.ClusterStateMutator building a new cName: 
control_collection
   [junit4]   2> 3581040 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard 
shard=shard1
   [junit4]   2> 3581042 INFO  
(zkCallback-2842-thread-1-processing-n:127.0.0.1:47355_) [n:127.0.0.1:47355_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred 
- updating... (live nodes size: [1])
   [junit4]   2> 3581042 INFO  
(zkCallback-2842-thread-2-processing-n:127.0.0.1:47355_) [n:127.0.0.1:47355_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred 
- updating... (live nodes size: [1])
   [junit4]   2> 3582038 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.c.ZkController 
waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3582039 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.c.ZkController 
Check for collection zkNode:control_collection
   [junit4]   2> 3582039 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.c.ZkController 
Collection zkNode exists
   [junit4]   2> 3582039 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config from: 
[/collections/control_collection]
   [junit4]   2> 3582040 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader path=[/collections/control_collection] 
[configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 3582040 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/control-001/cores/collection1'
   [junit4]   2> 3582040 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3582040 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 3582042 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.c.Config loaded 
config solrconfig.xml with version 0 
   [junit4]   2> 3582048 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
current version of requestparams : -1
   [junit4]   2> 3582052 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 3582062 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3582069 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 3582217 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
default search field in schema is text
   [junit4]   2> 3582218 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
unique key field: id
   [junit4]   2> 3582229 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 3582232 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 3582261 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] 
o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default 
properties: Can't find resource 'configsetprops.json' in classpath or 
'/configs/conf1', 
cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 3582261 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection   x:collection1] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1' using configuration from collection 
control_collection
   [junit4]   2> 3582261 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 3582261 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:52056/solr_hdfs_home
   [junit4]   2> 3582261 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3582262 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/control-001/cores/collection1],
 
dataDir=[hdfs://localhost:52056/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 3582262 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@af76dac
   [junit4]   2> 3582262 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:52056/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 3582280 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.CachingDirectoryFactory return new directory for 
hdfs://localhost:52056/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 3582282 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore New index directory detected: old=null 
new=hdfs://localhost:52056/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 3582294 WARN  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore [collection1] Solr index directory 
'hdfs:/localhost:52056/solr_hdfs_home/control_collection/core_node1/data/index' 
doesn't exist. Creating new index...
   [junit4]   2> 3582294 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:52056/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 3582304 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 3582304 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 3582304 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3582319 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3582321 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.CachingDirectoryFactory return new directory for 
hdfs://localhost:52056/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 3582321 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=2, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.40380481239202115]
   [junit4]   2> 3582372 INFO  (IPC Server handler 5 on 52056) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48975 is 
added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, 
primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[[DISK]DS-135f82ae-10b6-424f-9c1e-20e1ae53c3eb:NORMAL:127.0.0.1:47902|RBW],
 
ReplicaUnderConstruction[[DISK]DS-4299d711-c066-4898-b227-aefb31f022c0:NORMAL:127.0.0.1:48975|FINALIZED]]}
 size 0
   [junit4]   2> 3582378 INFO  (IPC Server handler 6 on 52056) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:47902 is 
added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, 
primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[[DISK]DS-135f82ae-10b6-424f-9c1e-20e1ae53c3eb:NORMAL:127.0.0.1:47902|RBW],
 
ReplicaUnderConstruction[[DISK]DS-4299d711-c066-4898-b227-aefb31f022c0:NORMAL:127.0.0.1:48975|FINALIZED]]}
 size 0
   [junit4]   2> 3582380 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>        
commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:52056/solr_hdfs_home/control_collection/core_node1/data/index
 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@78786eaf); 
maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 3582380 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 3582383 INFO  
(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
Looking for old index directories to cleanup for core collection1 in 
hdfs://localhost:52056/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 3582385 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"nodistrib"
   [junit4]   2> 3582385 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"dedupe"
   [junit4]   2> 3582385 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 3582385 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"stored_sig"
   [junit4]   2> 3582386 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3582386 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"distrib-dup-test-chain-explicit"
   [junit4]   2> 3582386 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 3582386 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 3582386 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating 
implicit default
   [junit4]   2> 3582387 WARN  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 3582389 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 3582390 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 3582390 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 3582401 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 3582415 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.RequestHandlers Registered paths: 
/admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 3582417 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore Using default statsCache cache: 
org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 3582418 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3582419 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: dataDir= 
defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 
tlogDfsReplication=2
   [junit4]   2> 3582446 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3582446 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3582449 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@7667994e
   [junit4]   2> 3582463 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>        
commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:52056/solr_hdfs_home/control_collection/core_node1/data/index
 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@78786eaf); 
maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 3582463 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 3582463 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.s.SolrIndexSearcher Opening Searcher@4977c8b6[collection1] main
   [junit4]   2> 3582463 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config from: 
[/collections/control_collection]
   [junit4]   2> 3582464 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.ZkStateReader path=[/collections/control_collection] 
[configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 3582464 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the 
RestManager with znodeBase: /configs/conf1
   [junit4]   2> 3582464 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 3582464 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 3582464 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3582465 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage No data found for znode 
/configs/conf1/_rest_managed.json
   [junit4]   2> 3582465 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3582465 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 3582465 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3582466 INFO  
(searcherExecutor-6714-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
[collection1] Registered new searcher Searcher@4977c8b6[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3582466 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 3582466 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.VersionInfo Refreshing highest value of _version_ for 256 version 
buckets from index
   [junit4]   2> 3582466 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket 
highest value from index
   [junit4]   2> 3582466 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1521559459775643648
   [junit4]   2> 3582466 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Took 0.0ms to seed version buckets with highest version 
1521559459775643648
   [junit4]   2> 3582466 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 3582467 INFO  
(coreLoadExecutor-6713-thread-1-processing-n:127.0.0.1:47355_) 
[n:127.0.0.1:47355_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 3582467 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
Register replica - core:collection1 address:http://127.0.0.1:47355 
collection:control_collection shard:shard1
   [junit4]   2> 3582468 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.SolrZkClient makePath: 
/collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 3582473 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector 
Joined leadership election with path: 
/collections/control_collection/leader_elect/shard1/election/95097466067353604-core_node1-n_0000000000
   [junit4]   2> 3582474 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 
and weAreReplacement=false and leaderVoteWait=10000
   [junit4]   2> 3582476 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 3582478 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3582478 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3582478 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:47355/collection1/
   [junit4]   2> 3582478 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
Sync Success - now sync replicas to me
   [junit4]   2> 3582479 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
http://127.0.0.1:47355/collection1/ has no replicas
   [junit4]   2> 3582479 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 3582481 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 3582483 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:47355";,
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 3582484 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:47355/collection1/ shard1
   [junit4]   2> 3582542 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3582544 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3582545 INFO  (zkCallback-2844-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@384b08f7 
name:ZooKeeperConnection Watcher:127.0.0.1:59246/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3582545 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3582545 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3582545 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3582547 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection 
loss:false
   [junit4]   2> 3582585 INFO  (zkCallback-2844-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred 
- updating... (live nodes size: [1])
   [junit4]   2> 3582585 INFO  
(zkCallback-2842-thread-1-processing-n:127.0.0.1:47355_) [n:127.0.0.1:47355_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred 
- updating... (live nodes size: [1])
   [junit4]   2> 3582585 INFO  
(zkCallback-2842-thread-3-processing-n:127.0.0.1:47355_) [n:127.0.0.1:47355_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred 
- updating... (live nodes size: [1])
   [junit4]   2> 3582635 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
We are http://127.0.0.1:47355/collection1/ and leader is 
http://127.0.0.1:47355/collection1/
   [junit4]   2> 3582635 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47355
   [junit4]   2> 3582636 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
I am the leader, no recovery necessary
   [junit4]   2> 3582636 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
publishing state=active
   [junit4]   2> 3582636 INFO  
(coreZkRegister-6707-thread-1-processing-n:127.0.0.1:47355_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47355_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
numShards not found on descriptor - reading it from system property
   [junit4]   2> 3582637 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:47355";,
   [junit4]   2>   "node_name":"127.0.0.1:47355_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 3582638 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:47355";,
   [junit4]   2>   "node_name":"127.0.0.1:47355_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 3582742 INFO  (zkCallback-2844-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred 
- updating... (live nodes size: [1])
   [junit4]   2> 3582742 INFO  
(zkCallback-2842-thread-3-processing-n:127.0.0.1:47355_) [n:127.0.0.1:47355_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred 
- updating... (live nodes size: [1])
   [junit4]   2> 3582742 INFO  
(zkCallback-2842-thread-1-processing-n:127.0.0.1:47355_) [n:127.0.0.1:47355_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred 
- updating... (live nodes size: [1])
   [junit4]   2> 3583164 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/shard-1-001/cores/collection1
   [junit4]   2> 3583165 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/shard-1-001
   [junit4]   2> 3583166 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] o.e.j.s.Server 
jetty-9.3.6.v20151106
   [junit4]   2> 3583179 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@3f6e26ed{/,null,AVAILABLE}
   [junit4]   2> 3583180 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@1222a842{HTTP/1.1,[http/1.1]}{127.0.0.1:52423}
   [junit4]   2> 3583180 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] o.e.j.s.Server 
Started @3586384ms
   [junit4]   2> 3583180 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=hdfs://localhost:52056/hdfs__localhost_52056__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-trunk_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001_tempDir-002_jetty1,
 solrconfig=solrconfig.xml, hostContext=/, hostPort=52423, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/shard-1-001/cores}
   [junit4]   2> 3583180 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 3583180 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/shard-1-001'
   [junit4]   2> 3583180 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3583181 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 3583181 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3583185 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3583192 INFO  (zkCallback-2845-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@8304f8e name:ZooKeeperConnection 
Watcher:127.0.0.1:59246/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 3583192 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3583192 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3583199 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 3583200 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/shard-1-001/solr.xml
   [junit4]   2> 3583204 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.CorePropertiesLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/shard-1-001/cores
   [junit4]   2> 3583204 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.CoreContainer New CoreContainer 1120695613
   [junit4]   2> 3583205 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/shard-1-001]
   [junit4]   2> 3583205 WARN  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.CoreContainer Couldn't add files from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/shard-1-001/lib
 to classpath: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/shard-1-001/lib
   [junit4]   2> 3583206 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme 
: ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 
10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 
5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 3583207 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3583208 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3583208 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3583208 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59246/solr
   [junit4]   2> 3583208 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 3583208 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3583223 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3583227 INFO  (zkCallback-2848-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@1cf3234 name:ZooKeeperConnection 
Watcher:127.0.0.1:59246 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 3583227 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3583227 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3583233 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3583233 INFO  
(zkCallback-2849-thread-1-processing-n:127.0.0.1:52423_) [n:127.0.0.1:52423_    
] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@486f5829 
name:ZooKeeperConnection Watcher:127.0.0.1:59246/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3583234 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3583240 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3584245 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:52423_
   [junit4]   2> 3584245 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:52423_
   [junit4]   2> 3584247 INFO  
(zkCallback-2849-thread-1-processing-n:127.0.0.1:52423_) [n:127.0.0.1:52423_    
] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live 
nodes size: [1])
   [junit4]   2> 3584247 INFO  (zkCallback-2844-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live 
nodes size: [1])
   [junit4]   2> 3584247 INFO  
(zkCallback-2842-thread-3-processing-n:127.0.0.1:47355_) [n:127.0.0.1:47355_    
] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live 
nodes size: [1])
   [junit4]   2> 3584247 INFO  
(zkCallback-2842-thread-1-processing-n:127.0.0.1:47355_) [n:127.0.0.1:47355_    
] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live 
nodes size: [1])
   [junit4]   2> 3584248 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3584248 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.LeaderElector Joined leadership election with path: 
/overseer_elect/election/95097466067353608-127.0.0.1:52423_-n_0000000001
   [junit4]   2> 3584249 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.LeaderElector Watching path 
/overseer_elect/election/95097466067353604-127.0.0.1:47355_-n_0000000000 to 
know if I could be the leader
   [junit4]   2> 3584290 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for 
authorization module.
   [junit4]   2> 3584290 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 3584291 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.CorePropertiesLocator Looking for core definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/shard-1-001/cores
   [junit4]   2> 3584292 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, 
config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, 
configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, 
dataDir=data/, shard=}
   [junit4]   2> 3584292 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.CorePropertiesLocator Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/shard-1-001/cores/collection1
   [junit4]   2> 3584292 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 3584294 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.ZkController 
publishing state=down
   [junit4]   2> 3584294 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.ZkController 
numShards not found on descriptor - reading it from system property
   [junit4]   2> 3584295 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:52423";,
   [junit4]   2>   "node_name":"127.0.0.1:52423_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 3584296 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:52423";,
   [junit4]   2>   "node_name":"127.0.0.1:52423_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 3584297 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.o.ClusterStateMutator building a new cName: 
collection1
   [junit4]   2> 3584297 INFO  
(OverseerStateUpdate-95097466067353604-127.0.0.1:47355_-n_0000000000) 
[n:127.0.0.1:47355_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard 
shard=shard1
   [junit4]   2> 3584300 INFO  (zkCallback-2844-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred 
- updating... (live nodes size: [2])
   [junit4]   2> 3584300 INFO  
(zkCallback-2842-thread-1-processing-n:127.0.0.1:47355_) [n:127.0.0.1:47355_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred 
- updating... (live nodes size: [2])
   [junit4]   2> 3584300 INFO  
(zkCallback-2842-thread-3-processing-n:127.0.0.1:47355_) [n:127.0.0.1:47355_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred 
- updating... (live nodes size: [2])
   [junit4]   2> 3584301 INFO  
(zkCallback-2849-thread-1-processing-n:127.0.0.1:52423_) [n:127.0.0.1:52423_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred 
- updating... (live nodes size: [2])
   [junit4]   2> 3584304 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.ZkController 
Registering watch for collection collection1
   [junit4]   2> 3584304 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader 
addZkWatch [collection1]
   [junit4]   2> 3584305 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader 
Deleting data for [collection1]
   [junit4]   2> 3584305 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.ZkController look 
for our core node name
   [junit4]   2> 3584305 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.ZkController waiting 
to find shard id in clusterstate for collection1
   [junit4]   2> 3584305 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.ZkController Check 
for collection zkNode:collection1
   [junit4]   2> 3584306 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.ZkController 
Collection zkNode exists
   [junit4]   2> 3584306 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Load 
collection config from: [/collections/collection1]
   [junit4]   2> 3584307 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader 
path=[/collections/collection1] [configName]=[conf1] specified config exists in 
ZooKeeper
   [junit4]   2> 3584307 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.SolrResourceLoader 
new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001/shard-1-001/cores/collection1'
   [junit4]   2> 3584307 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.SolrResourceLoader 
JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3584307 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.SolrResourceLoader 
solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 3584318 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.Config loaded config 
solrconfig.xml with version 0 
   [junit4]   2> 3584327 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 3584327 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[B49EC28476BD3CBD]) [n:127.0.0.1:52423_    ] 
o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 3584349 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.SolrConfig current 
version of requestparams : -1
   [junit4]   2> 3584356 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using 
Lucene MatchVersion: 6.0.0
   [junit4]   2> 3584372 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.c.SolrConfig Loaded 
SolrConfig: solrconfig.xml
   [junit4]   2> 3584379 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 3584576 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.s.IndexSchema default 
search field in schema is text
   [junit4]   2> 3584577 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] o.a.s.s.IndexSchema unique 
key field: id
   [junit4]   2> 3584579 INFO  
(coreLoadExecutor-6724-thread-1-processing-n:127.0.0.1:52423_) 
[n:127.0.0.1:52423_ c:collection1   x:collection1] 
o.a.s.s.FileExchangeRateProvider 

[...truncated too long message...]

unit4]   2>     at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:227)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:212)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:461)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:212)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:592)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:145)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:822)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1720)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1699)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:198)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest.teardownClass(HdfsRecoveryZkTest.java:48)
   [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:497)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2>        ... 54 more
   [junit4]   2> 3601011 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[B49EC28476BD3CBD]-worker) [n:127.0.0.1:52423_ 
c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.SolrTestCaseJ4 
###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B49EC28476BD3CBD-001
   [junit4]   2> Dec 25, 2015 7:36:36 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): 
{rnd_b=PostingsFormat(name=MockRandom), 
_version_=PostingsFormat(name=LuceneFixedGap), 
a_t=PostingsFormat(name=LuceneFixedGap), a_i=PostingsFormat(name=MockRandom), 
id=PostingsFormat(name=MockRandom)}, docValues:{}, sim=ClassicSimilarity, 
locale=da_DK, timezone=Australia/Lindeman
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 
1.8.0_66 (64-bit)/cpus=4,threads=2,free=382876096,total=523239424
   [junit4]   2> NOTE: All tests run in this JVM: 
[TestExclusionRuleCollectionAccess, TestSolrDeletionPolicy1, 
CdcrVersionReplicationTest, TestReload, TestCollapseQParserPlugin, 
TestManagedResource, RuleEngineTest, OverseerTaskQueueTest, SuggesterTSTTest, 
HdfsChaosMonkeyNothingIsSafeTest, TestCloudSchemaless, 
TestSerializedLuceneMatchVersion, TestRawTransformer, 
TestPostingsSolrHighlighter, RegexBoostProcessorTest, ForceLeaderTest, 
StatsComponentTest, TestJmxMonitoredMap, TermVectorComponentDistributedTest, 
TestCloudInspectUtil, TestInfoStreamLogging, TestSearchPerf, 
VMParamsZkACLAndCredentialsProvidersTest, AtomicUpdatesTest, 
TestHighFrequencyDictionaryFactory, TestGroupingSearch, ExplicitHLLTest, 
SuggesterWFSTTest, TestTolerantSearch, TestDistributedMissingSort, 
TestPKIAuthenticationPlugin, TestPivotHelperCode, TestSortByMinMaxFunction, 
TestZkChroot, OverseerCollectionConfigSetProcessorTest, LeaderElectionTest, 
ScriptEngineTest, TestExactSharedStatsCache, 
BigEndianAscendingWordDeserializerTest, SmileWriterTest, FacetPivotSmallTest, 
TestNonDefinedSimilarityFactory, SolrCloudExampleTest, TestChildDocTransformer, 
DistribJoinFromCollectionTest, ShowFileRequestHandlerTest, 
TestSuggestSpellingConverter, TestElisionMultitermQuery, 
TestMiniSolrCloudClusterSSL, DeleteInactiveReplicaTest, 
BlockJoinFacetRandomTest, SimpleFacetsTest, RecoveryZkTest, 
IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestShortCircuitedRequests, 
TestFieldCollectionResource, TestFuzzyAnalyzedSuggestions, TestCryptoKeys, 
TestSolrIndexConfig, HighlighterConfigTest, 
StatelessScriptUpdateProcessorFactoryTest, TestBlendedInfixSuggestions, 
LeaderInitiatedRecoveryOnCommitTest, AddBlockUpdateTest, 
FieldAnalysisRequestHandlerTest, TestIndexSearcher, TestInitQParser, 
TestJsonFacets, TestReplicaProperties, TestScoreJoinQPNoScore, 
BadIndexSchemaTest, TestReversedWildcardFilterFactory, TestSolrJ, 
SuggesterTest, TestRandomMergePolicy, MultiThreadedOCPTest, 
TestClusterStateMutator, ActionThrottleTest, TestCollationField, 
TestMiniSolrCloudCluster, SearchHandlerTest, TestRecoveryHdfs, 
DirectUpdateHandlerTest, AutoCommitTest, 
DistributedQueryComponentCustomSortTest, TestManagedSchemaFieldTypeResource, 
TestSchemaResource, DateMathParserTest, BasicAuthIntegrationTest, 
ClusterStateTest, TestNoOpRegenerator, CursorMarkTest, TestUtils, 
ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, 
OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, BasicZkTest, 
FullSolrCloudDistribCmdsTest, TestReplicationHandler, CdcrRequestHandlerTest, 
CloudExitableDirectoryReaderTest, CollectionStateFormat2Test, 
CollectionTooManyReplicasTest, ConfigSetsAPITest, 
DeleteLastCustomShardedReplicaTest, DeleteShardTest, DistribCursorPagingTest, 
DistributedVersionInfoTest, MigrateRouteKeyTest, 
OverriddenZkACLAndCredentialsProvidersTest, OverseerRolesTest, 
OverseerStatusTest, RecoveryAfterSoftCommitTest, ReplicationFactorTest, 
RestartWhileUpdatingTest, HdfsChaosMonkeySafeLeaderTest, HdfsNNFailoverTest, 
HdfsRecoverLeaseTest, HdfsRecoveryZkTest]
   [junit4] Completed [457/571 (1!)] on J2 in 33.43s, 1 test, 1 error <<< 
FAILURES!

[...truncated 374 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:791:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:728:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1436:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:991:
 There were test failures: 571 suites (8 ignored), 2276 tests, 1 error, 33 
ignored (18 assumptions) [seed: B49EC28476BD3CBD]

Total time: 211 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to