Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/49/
1 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest Error Message: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] Stack Trace: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] at __randomizedtesting.SeedInfo.seed([29242E7382F60DE5]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNull(Assert.java:551) at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:256) at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 11462 lines...] [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest [junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/init-core-data-001 [junit4] 2> 1622341 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) [junit4] 2> 1622341 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: / [junit4] 1> Formatting using clusterid: testClusterID [junit4] 2> 1622425 WARN (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties [junit4] 2> 1622464 WARN (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j [junit4] 2> 1622480 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.m.log jetty-6.1.26 [junit4] 2> 1622503 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_54521_hdfs____5bi6zj/webapp [junit4] 2> 1622669 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet [junit4] 2> 1623164 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54521 [junit4] 2> 1623679 WARN (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j [junit4] 2> 1623680 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.m.log jetty-6.1.26 [junit4] 2> 1623711 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_56145_datanode____n49cwo/webapp [junit4] 2> 1624292 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet [junit4] 2> 1624637 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56145 [junit4] 2> 1624755 WARN (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j [junit4] 2> 1624759 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.m.log jetty-6.1.26 [junit4] 2> 1624777 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_47744_datanode____4odnjv/webapp [junit4] 2> 1624902 INFO (IPC Server handler 4 on 49687) [ ] BlockStateChange BLOCK* processReport: from storage DS-e3bc9d2d-969a-4203-af71-a1842fc40a48 node DatanodeRegistration(127.0.0.1, datanodeUuid=88ce246c-19d0-40b0-9a16-ec36695954ce, infoPort=56145, ipcPort=38013, storageInfo=lv=-56;cid=testClusterID;nsid=257023781;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs [junit4] 2> 1624902 INFO (IPC Server handler 4 on 49687) [ ] BlockStateChange BLOCK* processReport: from storage DS-fdb9ae2f-1257-415a-be25-5e0f2d9f1f64 node DatanodeRegistration(127.0.0.1, datanodeUuid=88ce246c-19d0-40b0-9a16-ec36695954ce, infoPort=56145, ipcPort=38013, storageInfo=lv=-56;cid=testClusterID;nsid=257023781;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs [junit4] 2> 1625055 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet [junit4] 2> 1625800 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47744 [junit4] 2> 1626152 INFO (IPC Server handler 5 on 49687) [ ] BlockStateChange BLOCK* processReport: from storage DS-28498c57-0104-4d22-96ea-47b607dcd51b node DatanodeRegistration(127.0.0.1, datanodeUuid=a650ff14-ef3c-416d-96f6-3d03c79cf86f, infoPort=47744, ipcPort=59950, storageInfo=lv=-56;cid=testClusterID;nsid=257023781;c=0), blocks: 0, hasStaleStorages: true, processing time: 0 msecs [junit4] 2> 1626152 INFO (IPC Server handler 5 on 49687) [ ] BlockStateChange BLOCK* processReport: from storage DS-8d49033e-ebb6-4f06-b44b-83f6619be14c node DatanodeRegistration(127.0.0.1, datanodeUuid=a650ff14-ef3c-416d-96f6-3d03c79cf86f, infoPort=47744, ipcPort=59950, storageInfo=lv=-56;cid=testClusterID;nsid=257023781;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs [junit4] 2> 1626290 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER [junit4] 2> 1626298 INFO (Thread-5745) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 1626298 INFO (Thread-5745) [ ] o.a.s.c.ZkTestServer Starting server [junit4] 2> 1626398 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.ZkTestServer start zk server on port:42830 [junit4] 2> 1626398 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 1626417 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 1626429 INFO (zkCallback-966-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1db9f962 name:ZooKeeperConnection Watcher:127.0.0.1:42830 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1626429 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 1626430 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 1626430 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr [junit4] 2> 1626437 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 1626437 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 1626495 INFO (zkCallback-967-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@126f1b46 name:ZooKeeperConnection Watcher:127.0.0.1:42830/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1626495 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 1626495 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 1626495 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1 [junit4] 2> 1626496 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards [junit4] 2> 1626497 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection [junit4] 2> 1626497 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards [junit4] 2> 1626504 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml [junit4] 2> 1626505 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml [junit4] 2> 1626506 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml [junit4] 2> 1626506 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml [junit4] 2> 1626507 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 1626507 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 1626510 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt [junit4] 2> 1626510 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt [junit4] 2> 1626511 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt [junit4] 2> 1626511 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt [junit4] 2> 1626511 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml [junit4] 2> 1626511 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml [junit4] 2> 1626512 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml [junit4] 2> 1626512 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml [junit4] 2> 1626513 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json [junit4] 2> 1626513 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json [junit4] 2> 1626513 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt [junit4] 2> 1626514 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt [junit4] 2> 1626520 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt [junit4] 2> 1626520 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt [junit4] 2> 1626521 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt [junit4] 2> 1626521 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt [junit4] 2> 1629060 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores/collection1 [junit4] 2> 1629062 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.e.j.s.Server jetty-9.3.8.v20160314 [junit4] 2> 1629064 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@724bbdc4{/,null,AVAILABLE} [junit4] 2> 1629073 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.e.j.s.ServerConnector Started ServerConnector@a5af8bb{HTTP/1.1,[http/1.1]}{127.0.0.1:54816} [junit4] 2> 1629073 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.e.j.s.Server Started @1631369ms [junit4] 2> 1629073 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:49687/hdfs__localhost_49687__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001_tempDir-002_control_data, hostContext=/, hostPort=54816, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores} [junit4] 2> 1629073 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93 [junit4] 2> 1629073 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001' [junit4] 2> 1629074 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx) [junit4] 2> 1629074 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI) [junit4] 2> 1629074 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 1629080 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 1629097 INFO (zkCallback-968-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7a502592 name:ZooKeeperConnection Watcher:127.0.0.1:42830/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1629097 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 1629098 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 1629099 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1629099 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/solr.xml [junit4] 2> 1629102 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores [junit4] 2> 1629102 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.CoreContainer New CoreContainer 1498200714 [junit4] 2> 1629103 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001] [junit4] 2> 1629103 WARN (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/lib [junit4] 2> 1629103 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false, [junit4] 2> 1629107 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 1629107 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 1629107 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 1629107 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42830/solr [junit4] 2> 1629107 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.ZkController zkHost includes chroot [junit4] 2> 1629107 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 1629110 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 1629114 INFO (zkCallback-971-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@41aa8fa name:ZooKeeperConnection Watcher:127.0.0.1:42830 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1629115 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 1629115 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 1629120 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$45/1163214024@4b5587e7 [junit4] 2> 1629125 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 1629137 INFO (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ba235b0 name:ZooKeeperConnection Watcher:127.0.0.1:42830/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1629137 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 1629138 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue [junit4] 2> 1629139 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work [junit4] 2> 1629141 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running [junit4] 2> 1629142 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed [junit4] 2> 1629143 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure [junit4] 2> 1629143 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes [junit4] 2> 1629144 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json [junit4] 2> 1629145 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json [junit4] 2> 1629145 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /security.json [junit4] 2> 1629146 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... [junit4] 2> 1629146 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0) [junit4] 2> 1629147 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect [junit4] 2> 1629148 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election [junit4] 2> 1629149 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.Overseer Overseer (id=null) closing [junit4] 2> 1629149 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95793786670874628-127.0.0.1:54816_-n_0000000000 [junit4] 2> 1629150 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54816_ [junit4] 2> 1629150 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader [junit4] 2> 1629150 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.Overseer Overseer (id=95793786670874628-127.0.0.1:54816_-n_0000000000) starting [junit4] 2> 1629152 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work [junit4] 2> 1629172 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000 [junit4] 2> 1629180 INFO (OverseerCollectionConfigSetProcessor-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations [junit4] 2> 1629180 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.Overseer Starting to work on the main queue [junit4] 2> 1629181 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0) [junit4] 2> 1629182 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54816_ [junit4] 2> 1629183 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:54816_ [junit4] 2> 1629184 INFO (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0]) [junit4] 2> 1629185 INFO (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1629198 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module. [junit4] 2> 1629199 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.CoreContainer No authentication plugin used. [junit4] 2> 1629199 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores [junit4] 2> 1629212 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=} [junit4] 2> 1629212 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores/collection1 [junit4] 2> 1629212 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions [junit4] 2> 1629222 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.ZkController publishing state=down [junit4] 2> 1629222 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property [junit4] 2> 1629222 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible [junit4] 2> 1629222 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [control_collection] [junit4] 2> 1629223 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection] [junit4] 2> 1629223 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.ZkController look for our core node name [junit4] 2> 1629223 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:54816", [junit4] 2> "node_name":"127.0.0.1:54816_", [junit4] 2> "numShards":"1", [junit4] 2> "state":"down", [junit4] 2> "shard":null, [junit4] 2> "collection":"control_collection", [junit4] 2> "operation":"state"} current state version: 0 [junit4] 2> 1629224 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={ [junit4] 2> "core":"collection1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:54816", [junit4] 2> "node_name":"127.0.0.1:54816_", [junit4] 2> "numShards":"1", [junit4] 2> "state":"down", [junit4] 2> "shard":null, [junit4] 2> "collection":"control_collection", [junit4] 2> "operation":"state"} [junit4] 2> 1629224 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection [junit4] 2> 1629224 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1 [junit4] 2> 1629225 INFO (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1]) [junit4] 2> 1629226 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1 [junit4] 2> 1629226 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection [junit4] 2> 1629226 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.ZkController Collection zkNode exists [junit4] 2> 1629226 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection] [junit4] 2> 1629226 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper [junit4] 2> 1629226 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores/collection1' [junit4] 2> 1629227 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx) [junit4] 2> 1629227 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI) [junit4] 2> 1629228 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 [junit4] 2> 1629241 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2 [junit4] 2> 1629241 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done [junit4] 2> 1629242 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1 [junit4] 2> 1629254 WARN (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead. [junit4] 2> 1629254 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0 [junit4] 2> 1629261 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml [junit4] 2> 1629264 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test [junit4] 2> 1629358 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.s.IndexSchema default search field in schema is text [junit4] 2> 1629359 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.s.IndexSchema unique key field: id [junit4] 2> 1629359 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml [junit4] 2> 1629361 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml [junit4] 2> 1629389 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2 [junit4] 2> 1629389 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection [junit4] 2> 1629389 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory [junit4] 2> 1629389 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49687/solr_hdfs_home [junit4] 2> 1629389 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled [junit4] 2> 1629390 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores/collection1], dataDir=[hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/] [junit4] 2> 1629390 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@180baf58 [junit4] 2> 1629390 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data [junit4] 2> 1629418 WARN (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again. [junit4] 2> 1634535 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data [junit4] 2> 1634536 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/index/ [junit4] 2> 1634552 WARN (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:49687/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index... [junit4] 2> 1634552 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/index [junit4] 2> 1634561 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true] [junit4] 2> 1634562 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes [junit4] 2> 1634574 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled [junit4] 2> 1634574 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/index [junit4] 2> 1634580 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=77.9541015625, floorSegmentMB=1.2529296875, forceMergeDeletesPctAllowed=22.82809990582344, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.786929286943679 [junit4] 2> 1634668 INFO (IPC Server handler 2 on 49687) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59582 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e3bc9d2d-969a-4203-af71-a1842fc40a48:NORMAL:127.0.0.1:32787|RBW], ReplicaUnderConstruction[[DISK]DS-28498c57-0104-4d22-96ea-47b607dcd51b:NORMAL:127.0.0.1:59582|FINALIZED]]} size 0 [junit4] 2> 1634672 INFO (IPC Server handler 1 on 49687) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:32787 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e3bc9d2d-969a-4203-af71-a1842fc40a48:NORMAL:127.0.0.1:32787|RBW], ReplicaUnderConstruction[[DISK]DS-28498c57-0104-4d22-96ea-47b607dcd51b:NORMAL:127.0.0.1:59582|FINALIZED]]} size 0 [junit4] 2> 1634674 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1 [junit4] 2> commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66dee619); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1} [junit4] 2> 1634674 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1 [junit4] 2> 1634688 INFO (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/ [junit4] 2> 1634690 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib" [junit4] 2> 1634690 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe" [junit4] 2> 1634690 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe" [junit4] 2> 1634690 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig" [junit4] 2> 1634690 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig" [junit4] 2> 1634690 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit" [junit4] 2> 1634690 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 1634691 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 1634691 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default [junit4] 2> 1634691 WARN (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}} [junit4] 2> 1634692 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60 [junit4] 2> 1634692 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60 [junit4] 2> 1634693 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60 [junit4] 2> 1634693 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60 [junit4] 2> 1634695 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file [junit4] 2> 1634695 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache [junit4] 2> 1634699 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog [junit4] 2> 1634699 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 1634699 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2 [junit4] 2> 1634715 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; [junit4] 2> 1634715 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 1634720 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=50.3974609375, floorSegmentMB=1.275390625, forceMergeDeletesPctAllowed=5.693172197839953, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0 [junit4] 2> 1634738 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1 [junit4] 2> commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66dee619); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1} [junit4] 2> 1634738 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1 [junit4] 2> 1634738 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@9712526[collection1] main] [junit4] 2> 1634738 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection] [junit4] 2> 1634739 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper [junit4] 2> 1634739 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1 [junit4] 2> 1634739 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1634739 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {} [junit4] 2> 1634739 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1634739 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json [junit4] 2> 1634739 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1634739 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources [junit4] 2> 1634739 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 1634740 INFO (searcherExecutor-2239-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@9712526[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1634740 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets [junit4] 2> 1634740 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index [junit4] 2> 1634740 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index [junit4] 2> 1634740 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1532700595575586816 [junit4] 2> 1634742 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1532700595575586816 [junit4] 2> 1634742 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1 [junit4] 2> 1634743 INFO (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1 [junit4] 2> 1634756 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:54816 collection:control_collection shard:shard1 [junit4] 2> 1634756 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election [junit4] 2> 1634757 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95793786670874628-core_node1-n_0000000000 [junit4] 2> 1634758 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=10000 [junit4] 2> 1634763 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. [junit4] 2> 1634763 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync [junit4] 2> 1634763 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54816/collection1/ [junit4] 2> 1634763 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me [junit4] 2> 1634763 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:54816/collection1/ has no replicas [junit4] 2> 1634763 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection"} current state version: 1 [junit4] 2> 1634763 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1 [junit4] 2> 1634763 INFO (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1]) [junit4] 2> 1634765 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95793786670874628-core_node1-n_0000000000 [junit4] 2> 1634766 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:54816/collection1/ shard1 [junit4] 2> 1634766 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "base_url":"http://127.0.0.1:54816", [junit4] 2> "core":"collection1", [junit4] 2> "state":"active"} current state version: 2 [junit4] 2> 1634867 INFO (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1]) [junit4] 2> 1634916 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:54816/collection1/ and leader is http://127.0.0.1:54816/collection1/ [junit4] 2> 1634916 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54816 [junit4] 2> 1634917 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary [junit4] 2> 1634917 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active [junit4] 2> 1634917 INFO (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property [junit4] 2> 1634919 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "dataDir":"hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:54816", [junit4] 2> "node_name":"127.0.0.1:54816_", [junit4] 2> "numShards":"1", [junit4] 2> "state":"active", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "operation":"state", [junit4] 2> "ulogDir":"hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/tlog"} current state version: 3 [junit4] 2> 1634919 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={ [junit4] 2> "core":"collection1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "dataDir":"hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:54816", [junit4] 2> "node_name":"127.0.0.1:54816_", [junit4] 2> "numShards":"1", [junit4] 2> "state":"active", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "operation":"state", [junit4] 2> "ulogDir":"hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/tlog"} [junit4] 2> 1635021 INFO (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1]) [junit4] 2> 1635243 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:42830/solr [junit4] 2> 1635243 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 1635249 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 1635260 INFO (zkCallback-974-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@19a5442d name:ZooKeeperConnection Watcher:127.0.0.1:42830/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1635260 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 1635260 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 1635260 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... [junit4] 2> 1635261 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1635261 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false [junit4] 2> 1635404 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/cores/collection1 [junit4] 2> 1635404 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001 [junit4] 2> 1635405 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.e.j.s.Server jetty-9.3.8.v20160314 [junit4] 2> 1635449 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@63be0ebd{/,null,AVAILABLE} [junit4] 2> 1635449 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.e.j.s.ServerConnector Started ServerConnector@5fdc1b04{HTTP/1.1,[http/1.1]}{127.0.0.1:42574} [junit4] 2> 1635449 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.e.j.s.Server Started @1637745ms [junit4] 2> 1635450 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:49687/hdfs__localhost_49687__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=42574, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/cores} [junit4] 2> 1635450 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93 [junit4] 2> 1635450 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001' [junit4] 2> 1635450 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx) [junit4] 2> 1635450 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI) [junit4] 2> 1635450 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 1635466 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 1635487 INFO (zkCallback-975-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2a1bba4 name:ZooKeeperConnection Watcher:127.0.0.1:42830/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1635487 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 1635487 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 1635494 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1635494 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/solr.xml [junit4] 2> 1635498 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/cores [junit4] 2> 1635498 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.CoreContainer New CoreContainer 788676558 [junit4] 2> 1635498 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001] [junit4] 2> 1635498 WARN (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/lib [junit4] 2> 1635499 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false, [junit4] 2> 1635500 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true [junit4] 2> 1635501 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 1635501 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 1635501 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42830/solr [junit4] 2> 1635501 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.ZkController zkHost includes chroot [junit4] 2> 1635501 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider [junit4] 2> 1635503 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 1635524 INFO (zkCallback-978-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2154d36b name:ZooKeeperConnection Watcher:127.0.0.1:42830 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1635683 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 1635684 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider [junit4] 2> 1635686 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$45/1163214024@595696bf [junit4] 2> 1635699 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper [junit4] 2> 1635718 INFO (zkCallback-979-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2fea5e95 name:ZooKeeperConnection Watcher:127.0.0.1:42830/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1635719 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper [junit4] 2> 1635727 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... [junit4] 2> 1635728 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1635730 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.Overseer Overseer (id=null) closing [junit4] 2> 1635732 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95793786670874632-127.0.0.1:42574_-n_0000000001 [junit4] 2> 1635736 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95793786670874628-127.0.0.1:54816_-n_0000000000 to know if I could be the leader [junit4] 2> 1635737 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.ZkController Publish node=127.0.0.1:42574_ as DOWN [junit4] 2> 1635738 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"downnode", [junit4] 2> "node_name":"127.0.0.1:42574_"} current state version: 4 [junit4] 2> 1635738 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:42574_ [junit4] 2> 1635840 INFO (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1]) [junit4] 2> 1635840 INFO (zkCallback-979-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1]) [junit4] 2> 1635840 INFO (zkCallback-974-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1]) [junit4] 2> 1636739 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42574_ [junit4] 2> 1636739 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:42574_ [junit4] 2> 1636740 INFO (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1]) [junit4] 2> 1636740 INFO (zkCallback-979-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1]) [junit4] 2> 1636740 INFO (zkCallback-974-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1]) [junit4] 2> 1636740 INFO (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1636740 INFO (zkCallback-979-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1636740 INFO (zkCallback-974-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1636764 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module. [junit4] 2> 1636764 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.CoreContainer No authentication plugin used. [junit4] 2> 1636764 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/cores [junit4] 2> 1636765 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, dataDir=data/, shard=} [junit4] 2> 1636765 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/cores/collection1 [junit4] 2> 1636765 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions [junit4] 2> 1636774 INFO (coreLoadExecutor-2249-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ c:collection1 x:collection1] o.a.s.c.ZkController publishing state=down [junit4] 2> 1636774 INFO (coreLoadExecutor-2249-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ c:collection1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property [junit4] 2> 1636775 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:42574", [junit4] 2> "node_name":"127.0.0.1:42574_", [junit4] 2> "numShards":"2", [junit4] 2> "state":"down", [junit4] 2> "shard":null, [junit4] 2> "collection":"collection1", [junit4] 2> "operation":"state"} current state version: 5 [junit4] 2> 1636775 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={ [junit4] 2> "core":"collection1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:42574", [junit4] 2> "node_name":"127.0.0.1:42574_", [junit4] 2> "numShards":"2", [junit4] 2> "state":"down", [junit4] 2> "shard":null, [junit4] 2> "collection":"collection1", [junit4] 2> "operation":"state"} [junit4] 2> 1636775 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1 [junit4] 2> 1636775 INFO (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2 [junit4] 2> 1636776 INFO (coreLoadExecutor-2249-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ c:collection1 x:collection1] o.a.s.c.ZkController Collection collection1 not visible yet, but flagging it so a watch is registered when it becomes visible [junit4] 2> 1636776 INFO (coreLoadExecutor-2249-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ c:collection1 x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [collection1] [junit4] 2> 1636776 INFO (coreLoadExecutor-2249-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ c:collection1 x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [collection1] [junit4] 2> 1636776 INFO (coreLoadExecutor-2249-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ c:collection1 x:collection1] o.a.s.c.ZkController look for our core node name [junit4] 2> 1636779 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2 [junit4] 2> 1636779 INFO (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done [junit4] 2> 1636876 INFO (zkCallback-974-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2]) [junit4] 2> 1636877 INFO (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2]) [junit4] 2> 1636876 INFO (zkCallback-979-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ ] o.a.s.c.c.ZkStateR [...truncated too long message...] reAfterMaxFailures.java:64) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> Caused by: java.lang.NullPointerException [junit4] 2> at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198) [junit4] 2> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291) [junit4] 2> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223) [junit4] 2> ... 54 more [junit4] 2> 1784808 INFO (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore [junit4] 2> ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] [junit4] 2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException [junit4] 2> at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42) [junit4] 2> at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:130) [junit4] 2> at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:74) [junit4] 2> at org.apache.solr.update.HdfsUpdateLog.ensureLog(HdfsUpdateLog.java:308) [junit4] 2> at org.apache.solr.update.UpdateLog.deleteByQuery(UpdateLog.java:490) [junit4] 2> at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:467) [junit4] 2> at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:79) [junit4] 2> at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:52) [junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:943) [junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDeleteByQuery(DistributedUpdateProcessor.java:1438) [junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:1331) [junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:1140) [junit4] 2> at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:124) [junit4] 2> at org.apache.solr.handler.loader.JavabinLoader.delete(JavabinLoader.java:159) [junit4] 2> at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:112) [junit4] 2> at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:54) [junit4] 2> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) [junit4] 2> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69) [junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) [junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2015) [junit4] 2> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652) [junit4] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460) [junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229) [junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) [junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109) [junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) [junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) [junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160) [junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) [junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092) [junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399) [junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:518) [junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) [junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) [junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246) [junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) [junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) [junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> Apr 26, 2016 7:02:24 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks [junit4] 2> WARNING: Will linger awaiting termination of 2 leaked thread(s). [junit4] 2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarity(queryNorm=true,coord=yes): {}, locale=fr-BE, timezone=America/Indiana/Vevay [junit4] 2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=2,free=269967712,total=524812288 [junit4] 2> NOTE: All tests run in this JVM: [TestXIncludeConfig, LukeRequestHandlerTest, TestSchemaSimilarityResource, TestIntervalFaceting, DebugComponentTest, HdfsSyncSliceTest, DistributedVersionInfoTest, XmlUpdateRequestHandlerTest, StatsComponentTest, TestDocBasedVersionConstraints, TestWriterPerf, RequestHandlersTest, SparseHLLTest, TestCollationField, FullSolrCloudDistribCmdsTest, CurrencyFieldXmlFileTest, ConnectionManagerTest, TestLeaderElectionZkExpiry, DocValuesMultiTest, TestSolrCLIRunExample, TestCharFilters, SearchHandlerTest, DeleteInactiveReplicaTest, BaseCdcrDistributedZkTest, SuggesterTest, TestSolrIndexConfig, TestCloudManagedSchema, ClassificationUpdateProcessorFactoryTest, OverseerTest, CdcrRequestHandlerTest, DistributedFacetPivotWhiteBoxTest, SimpleFacetsTest, DistributedSuggestComponentTest, TlogReplayBufferedWhileIndexingTest, TestIndexingPerformance, ShardSplitTest, CollectionsAPISolrJTest, HdfsChaosMonkeySafeLeaderTest] [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=HdfsChaosMonkeySafeLeaderTest -Dtests.seed=29242E7382F60DE5 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=fr-BE -Dtests.timezone=America/Indiana/Vevay -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] ERROR 0.00s J2 | HdfsChaosMonkeySafeLeaderTest (suite) <<< [junit4] > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] [junit4] > at __randomizedtesting.SeedInfo.seed([29242E7382F60DE5]:0) [junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:256) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] Completed [150/591 (1!)] on J2 in 172.03s, 1 test, 1 failure <<< FAILURES! [...truncated 1401 lines...] BUILD FAILED /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 591 suites (10 ignored), 2381 tests, 1 suite-level error, 31 ignored (16 assumptions) [seed: 29242E7382F60DE5] Total time: 215 minutes 46 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts No prior successful build to compare, so performing full copy of artifacts Recording test results Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org