[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 269 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/269/

3 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
There were too many update fails (68 > 60) - we expect it can happen, but 
shouldn't easily

Stack Trace:
java.lang.AssertionError: There were too many update fails (68 > 60) - we 
expect it can happen, but shouldn't easily
        at 
__randomizedtesting.SeedInfo.seed([3BD7CAACFABD75FE:B383F57654411806]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertFalse(Assert.java:68)
        at 
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:218)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1485619511385,generation=2,filelist=[_2x.cfe, _2x.cfs, 
_2x.si, _5u.cfe, _5u.cfs, _5u.si, _8r.cfe, _8r.cfs, _8r.si, _bo.cfe, _bo.cfs, 
_bo.si, _el.cfe, _el.cfs, _el.si, _hi.cfe, _hi.cfs, _hi.si, _kf.cfe, _kf.cfs, 
_kf.si, _nc.cfe, _nc.cfs, _nc.si, _q9.cfe, _q9.cfs, _q9.si, _t6.cfe, _t6.cfs, 
_t6.si, _vc.cfe, _vc.cfs, _vc.si, _vd.cfe, _vd.cfs, _vd.si, _ve.cfe, _ve.cfs, 
_ve.si, _vf.cfe, _vf.cfs, _vf.si, _vg.cfe, _vg.cfs, _vg.si, _vh.cfe, _vh.cfs, 
_vh.si, _vi.cfe, _vi.cfs, _vi.si, _vj.cfe, _vj.cfs, _vj.si, _vk.cfe, _vk.cfs, 
_vk.si, _vl.cfe, _vl.cfs, _vl.si, _vm.cfe, _vm.cfs, _vm.si, _vn.cfe, _vn.cfs, 
_vn.si, _vo.cfe, _vo.cfs, _vo.si, _vp.cfe, _vp.cfs, _vp.si, _vq.cfe, _vq.cfs, 
_vq.si, _vr.cfe, _vr.cfs, _vr.si, _vs.cfe, _vs.cfs, _vs.si, _vt.cfe, _vt.cfs, 
_vt.si, _vu.cfe, _vu.cfs, _vu.si, _vv.cfe, _vv.cfs, _vv.si, _vw.cfe, _vw.cfs, 
_vw.si, _vx.cfe, _vx.cfs, _vx.si, _vy.cfe, _vy.cfs, _vy.si, _vz.cfe, _vz.cfs, 
_vz.si, _w0.cfe, _w0.cfs, _w0.si, _w1.cfe, _w1.cfs, _w1.si, _w2.cfe, _w2.cfs, 
_w2.si, segments_2]}]> but 
was:<[{indexVersion=1485619511385,generation=2,filelist=[_2x.cfe, _2x.cfs, 
_2x.si, _5u.cfe, _5u.cfs, _5u.si, _8r.cfe, _8r.cfs, _8r.si, _bo.cfe, _bo.cfs, 
_bo.si, _el.cfe, _el.cfs, _el.si, _hi.cfe, _hi.cfs, _hi.si, _kf.cfe, _kf.cfs, 
_kf.si, _nc.cfe, _nc.cfs, _nc.si, _q9.cfe, _q9.cfs, _q9.si, _t6.cfe, _t6.cfs, 
_t6.si, _vc.cfe, _vc.cfs, _vc.si, _vd.cfe, _vd.cfs, _vd.si, _ve.cfe, _ve.cfs, 
_ve.si, _vf.cfe, _vf.cfs, _vf.si, _vg.cfe, _vg.cfs, _vg.si, _vh.cfe, _vh.cfs, 
_vh.si, _vi.cfe, _vi.cfs, _vi.si, _vj.cfe, _vj.cfs, _vj.si, _vk.cfe, _vk.cfs, 
_vk.si, _vl.cfe, _vl.cfs, _vl.si, _vm.cfe, _vm.cfs, _vm.si, _vn.cfe, _vn.cfs, 
_vn.si, _vo.cfe, _vo.cfs, _vo.si, _vp.cfe, _vp.cfs, _vp.si, _vq.cfe, _vq.cfs, 
_vq.si, _vr.cfe, _vr.cfs, _vr.si, _vs.cfe, _vs.cfs, _vs.si, _vt.cfe, _vt.cfs, 
_vt.si, _vu.cfe, _vu.cfs, _vu.si, _vv.cfe, _vv.cfs, _vv.si, _vw.cfe, _vw.cfs, 
_vw.si, _vx.cfe, _vx.cfs, _vx.si, _vy.cfe, _vy.cfs, _vy.si, _vz.cfe, _vz.cfs, 
_vz.si, _w0.cfe, _w0.cfs, _w0.si, _w1.cfe, _w1.cfs, _w1.si, _w2.cfe, _w2.cfs, 
_w2.si, segments_2]}, 
{indexVersion=1485619511385,generation=3,filelist=[_2x.cfe, _2x.cfs, _2x.si, 
_5u.cfe, _5u.cfs, _5u.si, _8r.cfe, _8r.cfs, _8r.si, _bo.cfe, _bo.cfs, _bo.si, 
_el.cfe, _el.cfs, _el.si, _hi.cfe, _hi.cfs, _hi.si, _kf.cfe, _kf.cfs, _kf.si, 
_nc.cfe, _nc.cfs, _nc.si, _q9.cfe, _q9.cfs, _q9.si, _t6.cfe, _t6.cfs, _t6.si, 
_w2.cfe, _w2.cfs, _w2.si, _w3.cfe, _w3.cfs, _w3.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: 
expected:<[{indexVersion=1485619511385,generation=2,filelist=[_2x.cfe, _2x.cfs, 
_2x.si, _5u.cfe, _5u.cfs, _5u.si, _8r.cfe, _8r.cfs, _8r.si, _bo.cfe, _bo.cfs, 
_bo.si, _el.cfe, _el.cfs, _el.si, _hi.cfe, _hi.cfs, _hi.si, _kf.cfe, _kf.cfs, 
_kf.si, _nc.cfe, _nc.cfs, _nc.si, _q9.cfe, _q9.cfs, _q9.si, _t6.cfe, _t6.cfs, 
_t6.si, _vc.cfe, _vc.cfs, _vc.si, _vd.cfe, _vd.cfs, _vd.si, _ve.cfe, _ve.cfs, 
_ve.si, _vf.cfe, _vf.cfs, _vf.si, _vg.cfe, _vg.cfs, _vg.si, _vh.cfe, _vh.cfs, 
_vh.si, _vi.cfe, _vi.cfs, _vi.si, _vj.cfe, _vj.cfs, _vj.si, _vk.cfe, _vk.cfs, 
_vk.si, _vl.cfe, _vl.cfs, _vl.si, _vm.cfe, _vm.cfs, _vm.si, _vn.cfe, _vn.cfs, 
_vn.si, _vo.cfe, _vo.cfs, _vo.si, _vp.cfe, _vp.cfs, _vp.si, _vq.cfe, _vq.cfs, 
_vq.si, _vr.cfe, _vr.cfs, _vr.si, _vs.cfe, _vs.cfs, _vs.si, _vt.cfe, _vt.cfs, 
_vt.si, _vu.cfe, _vu.cfs, _vu.si, _vv.cfe, _vv.cfs, _vv.si, _vw.cfe, _vw.cfs, 
_vw.si, _vx.cfe, _vx.cfs, _vx.si, _vy.cfe, _vy.cfs, _vy.si, _vz.cfe, _vz.cfs, 
_vz.si, _w0.cfe, _w0.cfs, _w0.si, _w1.cfe, _w1.cfs, _w1.si, _w2.cfe, _w2.cfs, 
_w2.si, segments_2]}]> but 
was:<[{indexVersion=1485619511385,generation=2,filelist=[_2x.cfe, _2x.cfs, 
_2x.si, _5u.cfe, _5u.cfs, _5u.si, _8r.cfe, _8r.cfs, _8r.si, _bo.cfe, _bo.cfs, 
_bo.si, _el.cfe, _el.cfs, _el.si, _hi.cfe, _hi.cfs, _hi.si, _kf.cfe, _kf.cfs, 
_kf.si, _nc.cfe, _nc.cfs, _nc.si, _q9.cfe, _q9.cfs, _q9.si, _t6.cfe, _t6.cfs, 
_t6.si, _vc.cfe, _vc.cfs, _vc.si, _vd.cfe, _vd.cfs, _vd.si, _ve.cfe, _ve.cfs, 
_ve.si, _vf.cfe, _vf.cfs, _vf.si, _vg.cfe, _vg.cfs, _vg.si, _vh.cfe, _vh.cfs, 
_vh.si, _vi.cfe, _vi.cfs, _vi.si, _vj.cfe, _vj.cfs, _vj.si, _vk.cfe, _vk.cfs, 
_vk.si, _vl.cfe, _vl.cfs, _vl.si, _vm.cfe, _vm.cfs, _vm.si, _vn.cfe, _vn.cfs, 
_vn.si, _vo.cfe, _vo.cfs, _vo.si, _vp.cfe, _vp.cfs, _vp.si, _vq.cfe, _vq.cfs, 
_vq.si, _vr.cfe, _vr.cfs, _vr.si, _vs.cfe, _vs.cfs, _vs.si, _vt.cfe, _vt.cfs, 
_vt.si, _vu.cfe, _vu.cfs, _vu.si, _vv.cfe, _vv.cfs, _vv.si, _vw.cfe, _vw.cfs, 
_vw.si, _vx.cfe, _vx.cfs, _vx.si, _vy.cfe, _vy.cfs, _vy.si, _vz.cfe, _vz.cfs, 
_vz.si, _w0.cfe, _w0.cfs, _w0.si, _w1.cfe, _w1.cfs, _w1.si, _w2.cfe, _w2.cfs, 
_w2.si, segments_2]}, 
{indexVersion=1485619511385,generation=3,filelist=[_2x.cfe, _2x.cfs, _2x.si, 
_5u.cfe, _5u.cfs, _5u.si, _8r.cfe, _8r.cfs, _8r.si, _bo.cfe, _bo.cfs, _bo.si, 
_el.cfe, _el.cfs, _el.si, _hi.cfe, _hi.cfs, _hi.si, _kf.cfe, _kf.cfs, _kf.si, 
_nc.cfe, _nc.cfs, _nc.si, _q9.cfe, _q9.cfs, _q9.si, _t6.cfe, _t6.cfs, _t6.si, 
_w2.cfe, _w2.cfs, _w2.si, _w3.cfe, _w3.cfs, _w3.si, segments_3]}]>
        at 
__randomizedtesting.SeedInfo.seed([3BD7CAACFABD75FE:1E00D19C8AF57BFD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at 
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1245)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
__randomizedtesting.SeedInfo.seed([3BD7CAACFABD75FE:B383F57654411806]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at 
org.apache.solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest.test(HdfsWriteToMultipleCollectionsTest.java:137)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11717 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/init-core-data-001
   [junit4]   2> 930635 INFO  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 930635 INFO  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: 
/_rsb/de
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 930674 WARN  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 930691 WARN  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 930693 INFO  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 930716 INFO  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_55755_hdfs____vsj95t/webapp
   [junit4]   2> 931147 INFO  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:55755
   [junit4]   2> 931218 WARN  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 931223 INFO  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 931238 INFO  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_49331_datanode____.hzlsdp/webapp
   [junit4]   2> 931676 INFO  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:49331
   [junit4]   2> 931791 WARN  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 931792 INFO  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.m.log jetty-6.1.26
   [junit4]   2> 931810 INFO  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_58030_datanode____dx7coh/webapp
   [junit4]   2> 931887 INFO  (IPC Server handler 4 on 37858) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-c4c902f9-e6fb-4a77-83c0-4099bccf5c0e node 
DatanodeRegistration(127.0.0.1:50157, 
datanodeUuid=254f28a6-06dc-4724-a708-c271c64dd987, infoPort=52595, 
infoSecurePort=0, ipcPort=47979, 
storageInfo=lv=-56;cid=testClusterID;nsid=1015618064;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 931888 INFO  (IPC Server handler 4 on 37858) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-30339dcb-6bc9-46f5-8393-d4b23aae8f2a node 
DatanodeRegistration(127.0.0.1:50157, 
datanodeUuid=254f28a6-06dc-4724-a708-c271c64dd987, infoPort=52595, 
infoSecurePort=0, ipcPort=47979, 
storageInfo=lv=-56;cid=testClusterID;nsid=1015618064;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 932227 INFO  
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[3BD7CAACFABD75FE]-worker) [    ] 
o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:58030
   [junit4]   2> 932388 INFO  (IPC Server handler 8 on 37858) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-0fe0fe47-f40a-4383-8649-a2417129c4fb node 
DatanodeRegistration(127.0.0.1:35099, 
datanodeUuid=80094b17-95a8-4e5b-9491-a5eeb3649e6e, infoPort=50236, 
infoSecurePort=0, ipcPort=50841, 
storageInfo=lv=-56;cid=testClusterID;nsid=1015618064;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 932389 INFO  (IPC Server handler 8 on 37858) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-9e565227-96fe-444a-a993-1ce9dc7f705a node 
DatanodeRegistration(127.0.0.1:35099, 
datanodeUuid=80094b17-95a8-4e5b-9491-a5eeb3649e6e, infoPort=50236, 
infoSecurePort=0, ipcPort=50841, 
storageInfo=lv=-56;cid=testClusterID;nsid=1015618064;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 932509 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 932510 INFO  (Thread-8327) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 932510 INFO  (Thread-8327) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 932610 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:53542
   [junit4]   2> 932624 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 932626 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 932628 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 932629 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 932631 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 932633 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 932635 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 932636 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 932638 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 932640 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 932642 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 932749 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/control-001/cores/collection1
   [junit4]   2> 932751 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 932753 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@4e08b035{/_rsb/de,null,AVAILABLE}
   [junit4]   2> 932753 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@7ea224f9{HTTP/1.1,[http/1.1]}{127.0.0.1:53126}
   [junit4]   2> 932753 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.Server Started @936849ms
   [junit4]   2> 932753 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=hdfs://localhost:37858/hdfs__localhost_37858__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001_tempDir-002_control_data,
 hostContext=/_rsb/de, hostPort=53126, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/control-001/cores}
   [junit4]   2> 932759 ERROR 
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 932759 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.5.0
   [junit4]   2> 932759 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 932760 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 932760 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-01-28T16:01:56.248Z
   [junit4]   2> 932767 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 932767 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/control-001/solr.xml
   [junit4]   2> 932774 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 932774 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53542/solr
   [junit4]   2> 932793 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.OverseerElectionContext I am going to 
be the leader 127.0.0.1:53126__rsb%2Fde
   [junit4]   2> 932794 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.Overseer Overseer 
(id=97361547493310468-127.0.0.1:53126__rsb%2Fde-n_0000000000) starting
   [junit4]   2> 932805 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:53126__rsb%2Fde
   [junit4]   2> 932807 INFO  
(zkCallback-1126-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (0) -> (1)
   [junit4]   2> 932834 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.CorePropertiesLocator Found 1 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/control-001/cores
   [junit4]   2> 932834 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.CorePropertiesLocator Cores are: 
[collection1]
   [junit4]   2> 932850 INFO  
(OverseerStateUpdate-97361547493310468-127.0.0.1:53126__rsb%2Fde-n_0000000000) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.o.ReplicaMutator Assigning new node 
to shard shard=shard1
   [junit4]   2> 933867 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 933883 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 934508 WARN  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.s.IndexSchema [collection1] default search field in schema is text. 
WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 934510 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 934529 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from 
collection control_collection
   [junit4]   2> 934530 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:37858/solr_hdfs_home
   [junit4]   2> 934530 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 934530 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/control-001/cores/collection1],
 
dataDir=[hdfs://localhost:37858/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 934530 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@55f13e17
   [junit4]   2> 934531 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:37858/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 934539 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 934539 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 934539 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 934706 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 934707 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:37858/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 934752 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:37858/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 934763 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 934763 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 934769 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 934770 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.1487613265856859]
   [junit4]   2> 934811 INFO  (IPC Server handler 7 on 37858) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50157 is 
added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-9e565227-96fe-444a-a993-1ce9dc7f705a:NORMAL:127.0.0.1:35099|RBW],
 
ReplicaUC[[DISK]DS-30339dcb-6bc9-46f5-8393-d4b23aae8f2a:NORMAL:127.0.0.1:50157|FINALIZED]]}
 size 0
   [junit4]   2> 934817 INFO  (IPC Server handler 6 on 37858) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35099 is 
added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-9e565227-96fe-444a-a993-1ce9dc7f705a:NORMAL:127.0.0.1:35099|RBW],
 
ReplicaUC[[DISK]DS-30339dcb-6bc9-46f5-8393-d4b23aae8f2a:NORMAL:127.0.0.1:50157|FINALIZED]]}
 size 0
   [junit4]   2> 934828 WARN  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 934839 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 934839 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 934840 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 934849 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 934849 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 934851 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=5, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=40.2119140625, 
floorSegmentMB=1.5556640625, forceMergeDeletesPctAllowed=20.293514544785463, 
segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 934862 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@7b6a61c4[collection1] main]
   [junit4]   2> 934863 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 934864 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 934864 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 934866 INFO  
(searcherExecutor-2093-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde 
x:collection1 c:control_collection) [n:127.0.0.1:53126__rsb%2Fde 
c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered 
new searcher Searcher@7b6a61c4[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 934869 INFO  
(coreLoadExecutor-2092-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde c:control_collection   x:collection1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1557784762365509632
   [junit4]   2> 934882 INFO  
(coreZkRegister-2085-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde 
x:collection1 c:control_collection) [n:127.0.0.1:53126__rsb%2Fde 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 934882 INFO  
(coreZkRegister-2085-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde 
x:collection1 c:control_collection) [n:127.0.0.1:53126__rsb%2Fde 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 934882 INFO  
(coreZkRegister-2085-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde 
x:collection1 c:control_collection) [n:127.0.0.1:53126__rsb%2Fde 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:53126/_rsb/de/collection1/
   [junit4]   2> 934882 INFO  
(coreZkRegister-2085-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde 
x:collection1 c:control_collection) [n:127.0.0.1:53126__rsb%2Fde 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
Sync Success - now sync replicas to me
   [junit4]   2> 934882 INFO  
(coreZkRegister-2085-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde 
x:collection1 c:control_collection) [n:127.0.0.1:53126__rsb%2Fde 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
http://127.0.0.1:53126/_rsb/de/collection1/ has no replicas
   [junit4]   2> 934891 INFO  
(coreZkRegister-2085-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde 
x:collection1 c:control_collection) [n:127.0.0.1:53126__rsb%2Fde 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:53126/_rsb/de/collection1/ shard1
   [junit4]   2> 934967 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 934974 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53542/solr ready
   [junit4]   2> 934974 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection 
loss:false
   [junit4]   2> 935042 INFO  
(coreZkRegister-2085-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde 
x:collection1 c:control_collection) [n:127.0.0.1:53126__rsb%2Fde 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
I am the leader, no recovery necessary
   [junit4]   2> 935076 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-1-001/cores/collection1
   [junit4]   2> 935076 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-1-001
   [junit4]   2> 935077 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 935079 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@4fb7492a{/_rsb/de,null,AVAILABLE}
   [junit4]   2> 935079 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@302330d6{HTTP/1.1,[http/1.1]}{127.0.0.1:50575}
   [junit4]   2> 935084 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.Server Started @939179ms
   [junit4]   2> 935084 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=hdfs://localhost:37858/hdfs__localhost_37858__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001_tempDir-002_jetty1,
 solrconfig=solrconfig.xml, hostContext=/_rsb/de, hostPort=50575, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-1-001/cores}
   [junit4]   2> 935084 ERROR 
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 935085 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.5.0
   [junit4]   2> 935085 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 935085 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 935085 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-01-28T16:01:58.573Z
   [junit4]   2> 935087 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 935088 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-1-001/solr.xml
   [junit4]   2> 935095 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 935096 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53542/solr
   [junit4]   2> 935106 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:50575__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (0) -> (1)
   [junit4]   2> 935112 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:50575__rsb%2Fde    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:50575__rsb%2Fde
   [junit4]   2> 935113 INFO  (zkCallback-1130-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 935113 INFO  
(zkCallback-1126-thread-2-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (1) -> (2)
   [junit4]   2> 935115 INFO  
(zkCallback-1135-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (1) -> (2)
   [junit4]   2> 935138 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:50575__rsb%2Fde    ] o.a.s.c.CorePropertiesLocator Found 1 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-1-001/cores
   [junit4]   2> 935138 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:50575__rsb%2Fde    ] o.a.s.c.CorePropertiesLocator Cores are: 
[collection1]
   [junit4]   2> 935140 INFO  
(OverseerStateUpdate-97361547493310468-127.0.0.1:53126__rsb%2Fde-n_0000000000) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.o.ReplicaMutator Assigning new node 
to shard shard=shard2
   [junit4]   2> 936156 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 936170 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 936242 WARN  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] default search field in schema is text. WARNING: Deprecated, 
please use 'df' on request instead.
   [junit4]   2> 936243 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] o.a.s.s.IndexSchema 
Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 936263 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from 
collection collection1
   [junit4]   2> 936263 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:37858/solr_hdfs_home
   [junit4]   2> 936263 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 936263 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] o.a.s.c.SolrCore 
[[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-1-001/cores/collection1],
 dataDir=[hdfs://localhost:37858/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 936263 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@55f13e17
   [junit4]   2> 936264 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:37858/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 936272 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 936272 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 936278 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 936279 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:37858/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 936298 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:37858/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 936304 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 936304 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 936307 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 936307 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.1487613265856859]
   [junit4]   2> 936326 INFO  (IPC Server handler 1 on 37858) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35099 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-30339dcb-6bc9-46f5-8393-d4b23aae8f2a:NORMAL:127.0.0.1:50157|RBW],
 
ReplicaUC[[DISK]DS-0fe0fe47-f40a-4383-8649-a2417129c4fb:NORMAL:127.0.0.1:35099|RBW]]}
 size 0
   [junit4]   2> 936328 INFO  (IPC Server handler 0 on 37858) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50157 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-0fe0fe47-f40a-4383-8649-a2417129c4fb:NORMAL:127.0.0.1:35099|RBW],
 
ReplicaUC[[DISK]DS-c4c902f9-e6fb-4a77-83c0-4099bccf5c0e:NORMAL:127.0.0.1:50157|FINALIZED]]}
 size 0
   [junit4]   2> 936344 WARN  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 936361 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 936361 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 936361 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 936371 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 936371 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 936373 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=5, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=40.2119140625, 
floorSegmentMB=1.5556640625, forceMergeDeletesPctAllowed=20.293514544785463, 
segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 936378 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@668a2a5[collection1] main]
   [junit4]   2> 936379 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 936380 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 936380 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 936381 INFO  
(searcherExecutor-2104-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:50575__rsb%2Fde c:collection1   
x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@668a2a5[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 936388 INFO  
(coreLoadExecutor-2103-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde c:collection1   x:collection1] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1557784763958296576
   [junit4]   2> 936396 INFO  
(coreZkRegister-2098-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:50575__rsb%2Fde c:collection1 
s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough 
replicas found to continue.
   [junit4]   2> 936396 INFO  
(coreZkRegister-2098-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:50575__rsb%2Fde c:collection1 
s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may 
be the new leader - try and sync
   [junit4]   2> 936396 INFO  
(coreZkRegister-2098-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:50575__rsb%2Fde c:collection1 
s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:50575/_rsb/de/collection1/
   [junit4]   2> 936396 INFO  
(coreZkRegister-2098-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:50575__rsb%2Fde c:collection1 
s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now 
sync replicas to me
   [junit4]   2> 936396 INFO  
(coreZkRegister-2098-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:50575__rsb%2Fde c:collection1 
s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
http://127.0.0.1:50575/_rsb/de/collection1/ has no replicas
   [junit4]   2> 936401 INFO  
(coreZkRegister-2098-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:50575__rsb%2Fde c:collection1 
s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am 
the new leader: http://127.0.0.1:50575/_rsb/de/collection1/ shard2
   [junit4]   2> 936552 INFO  
(coreZkRegister-2098-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:50575__rsb%2Fde c:collection1 
s:shard2 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no 
recovery necessary
   [junit4]   2> 936919 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-2-001/cores/collection1
   [junit4]   2> 936920 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-2-001
   [junit4]   2> 936921 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 936922 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@65758abb{/_rsb/de,null,AVAILABLE}
   [junit4]   2> 936923 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@7f01130f{HTTP/1.1,[http/1.1]}{127.0.0.1:43446}
   [junit4]   2> 936923 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.Server Started @941018ms
   [junit4]   2> 936923 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=hdfs://localhost:37858/hdfs__localhost_37858__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001_tempDir-002_jetty2,
 solrconfig=solrconfig.xml, hostContext=/_rsb/de, hostPort=43446, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-2-001/cores}
   [junit4]   2> 936923 ERROR 
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 936924 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.5.0
   [junit4]   2> 936924 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 936924 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 936924 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-01-28T16:02:00.412Z
   [junit4]   2> 936928 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 936928 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-2-001/solr.xml
   [junit4]   2> 936934 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 936935 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53542/solr
   [junit4]   2> 936945 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:43446__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (0) -> (2)
   [junit4]   2> 936951 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:43446__rsb%2Fde    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:43446__rsb%2Fde
   [junit4]   2> 936952 INFO  
(zkCallback-1135-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (2) -> (3)
   [junit4]   2> 936953 INFO  (zkCallback-1130-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 936954 INFO  
(zkCallback-1126-thread-1-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (2) -> (3)
   [junit4]   2> 936954 INFO  
(zkCallback-1141-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (2) -> (3)
   [junit4]   2> 937015 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:43446__rsb%2Fde    ] o.a.s.c.CorePropertiesLocator Found 1 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-2-001/cores
   [junit4]   2> 937015 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:43446__rsb%2Fde    ] o.a.s.c.CorePropertiesLocator Cores are: 
[collection1]
   [junit4]   2> 937022 INFO  
(OverseerStateUpdate-97361547493310468-127.0.0.1:53126__rsb%2Fde-n_0000000000) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.o.ReplicaMutator Assigning new node 
to shard shard=shard3
   [junit4]   2> 938035 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 938048 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 938124 WARN  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] default search field in schema is text. WARNING: Deprecated, 
please use 'df' on request instead.
   [junit4]   2> 938125 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] o.a.s.s.IndexSchema 
Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 938143 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from 
collection collection1
   [junit4]   2> 938143 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:37858/solr_hdfs_home
   [junit4]   2> 938143 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 938143 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] o.a.s.c.SolrCore 
[[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-2-001/cores/collection1],
 dataDir=[hdfs://localhost:37858/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 938144 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@55f13e17
   [junit4]   2> 938144 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:37858/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 938152 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 938152 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 938157 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 938157 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:37858/solr_hdfs_home/collection1/core_node2/data
   [junit4]   2> 938176 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:37858/solr_hdfs_home/collection1/core_node2/data/index
   [junit4]   2> 938183 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 938183 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 938185 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 938186 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.1487613265856859]
   [junit4]   2> 938198 INFO  (IPC Server handler 1 on 37858) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35099 is 
added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-30339dcb-6bc9-46f5-8393-d4b23aae8f2a:NORMAL:127.0.0.1:50157|RBW],
 
ReplicaUC[[DISK]DS-9e565227-96fe-444a-a993-1ce9dc7f705a:NORMAL:127.0.0.1:35099|RBW]]}
 size 0
   [junit4]   2> 938199 INFO  (IPC Server handler 0 on 37858) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50157 is 
added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-30339dcb-6bc9-46f5-8393-d4b23aae8f2a:NORMAL:127.0.0.1:50157|RBW],
 
ReplicaUC[[DISK]DS-9e565227-96fe-444a-a993-1ce9dc7f705a:NORMAL:127.0.0.1:35099|RBW]]}
 size 0
   [junit4]   2> 938206 WARN  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 938242 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 938242 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 938242 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 938254 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 938254 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 938256 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=5, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=40.2119140625, 
floorSegmentMB=1.5556640625, forceMergeDeletesPctAllowed=20.293514544785463, 
segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 938262 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@5f7200da[collection1] main]
   [junit4]   2> 938263 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 938263 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 938263 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 938265 INFO  
(searcherExecutor-2115-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:43446__rsb%2Fde c:collection1   
x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@5f7200da[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 938268 INFO  
(coreLoadExecutor-2114-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde c:collection1   x:collection1] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1557784765929619456
   [junit4]   2> 938276 INFO  
(coreZkRegister-2109-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:43446__rsb%2Fde c:collection1 
s:shard3 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough 
replicas found to continue.
   [junit4]   2> 938276 INFO  
(coreZkRegister-2109-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:43446__rsb%2Fde c:collection1 
s:shard3 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I may 
be the new leader - try and sync
   [junit4]   2> 938276 INFO  
(coreZkRegister-2109-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:43446__rsb%2Fde c:collection1 
s:shard3 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:43446/_rsb/de/collection1/
   [junit4]   2> 938276 INFO  
(coreZkRegister-2109-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:43446__rsb%2Fde c:collection1 
s:shard3 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync Success - now 
sync replicas to me
   [junit4]   2> 938276 INFO  
(coreZkRegister-2109-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:43446__rsb%2Fde c:collection1 
s:shard3 r:core_node2 x:collection1] o.a.s.c.SyncStrategy 
http://127.0.0.1:43446/_rsb/de/collection1/ has no replicas
   [junit4]   2> 938280 INFO  
(coreZkRegister-2109-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:43446__rsb%2Fde c:collection1 
s:shard3 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I am 
the new leader: http://127.0.0.1:43446/_rsb/de/collection1/ shard3
   [junit4]   2> 938431 INFO  
(coreZkRegister-2109-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde 
x:collection1 c:collection1) [n:127.0.0.1:43446__rsb%2Fde c:collection1 
s:shard3 r:core_node2 x:collection1] o.a.s.c.ZkController I am the leader, no 
recovery necessary
   [junit4]   2> 938789 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-3-001/cores/collection1
   [junit4]   2> 938790 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-3-001
   [junit4]   2> 938790 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 938792 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@695a419e{/_rsb/de,null,AVAILABLE}
   [junit4]   2> 938792 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@5a2f21e4{HTTP/1.1,[http/1.1]}{127.0.0.1:56634}
   [junit4]   2> 938792 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.e.j.s.Server Started @942888ms
   [junit4]   2> 938793 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=hdfs://localhost:37858/hdfs__localhost_37858__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001_tempDir-002_jetty3,
 solrconfig=solrconfig.xml, hostContext=/_rsb/de, hostPort=56634, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-3-001/cores}
   [junit4]   2> 938793 ERROR 
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 938794 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.5.0
   [junit4]   2> 938794 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 938794 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 938794 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-01-28T16:02:02.282Z
   [junit4]   2> 938798 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 938798 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-3-001/solr.xml
   [junit4]   2> 938804 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 938805 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53542/solr
   [junit4]   2> 938817 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:56634__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (0) -> (3)
   [junit4]   2> 938834 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:56634__rsb%2Fde    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:56634__rsb%2Fde
   [junit4]   2> 938837 INFO  
(zkCallback-1141-thread-1-processing-n:127.0.0.1:43446__rsb%2Fde) 
[n:127.0.0.1:43446__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (3) -> (4)
   [junit4]   2> 938838 INFO  
(zkCallback-1126-thread-3-processing-n:127.0.0.1:53126__rsb%2Fde) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (3) -> (4)
   [junit4]   2> 938837 INFO  (zkCallback-1130-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 938838 INFO  
(zkCallback-1147-thread-1-processing-n:127.0.0.1:56634__rsb%2Fde) 
[n:127.0.0.1:56634__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (3) -> (4)
   [junit4]   2> 938839 INFO  
(zkCallback-1135-thread-1-processing-n:127.0.0.1:50575__rsb%2Fde) 
[n:127.0.0.1:50575__rsb%2Fde    ] o.a.s.c.c.ZkStateReader Updated live nodes 
from ZooKeeper... (3) -> (4)
   [junit4]   2> 938922 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:56634__rsb%2Fde    ] o.a.s.c.CorePropertiesLocator Found 1 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_3BD7CAACFABD75FE-001/shard-3-001/cores
   [junit4]   2> 938922 INFO  
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[3BD7CAACFABD75FE]) 
[n:127.0.0.1:56634__rsb%2Fde    ] o.a.s.c.CorePropertiesLocator Cores are: 
[collection1]
   [junit4]   2> 938926 INFO  
(OverseerStateUpdate-97361547493310468-127.0.0.1:53126__rsb%2Fde-n_0000000000) 
[n:127.0.0.1:53126__rsb%2Fde    ] o.a.s.c.o.ReplicaMutator Assigning new node 
to shard shard=shard4
   [junit4]   2> 939937 INFO  
(coreLoadExecutor-2125-thread-1-processing-n:127.0.0.1:56634__rsb%2Fde) 
[n:127.0.0.1:56634__rsb%2Fde c:collection1   x:collection1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 939955 INFO  
(coreLoadExecutor-2125-thread-1-processing-n:127.0.0.1:56634__rsb%2Fde) 
[n:127.0.0.1:56634__rsb%2Fde c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 940041 WARN  
(coreLoadExecutor-2125-thread-1-processing-n:127.0.0.1:56634__rsb%2Fde) 
[n:127.0.0.1:56634__rsb%2Fde c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] default search field in schema is text. WARNING: Deprecated, 
please use 'df' on request instead.
   [junit4]   2> 940042 INFO  
(coreLoadExecutor-2125-thread-1-processing-n:127.0.0.1:56634__rsb%2Fde) 
[n:127.0.0.1:56634__rsb%2Fde c:collection1   x:collection1] o.a.s.s.IndexSchema 
Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 940060 INFO  
(coreLoadExecutor-2125-thread-1-processing-n:127.0.0.1:56634__rsb%2Fde) 
[n:127.0.0.1:56634__rsb%2Fde c:collection1   x:collection1] 
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from 
collection collection1
   [junit4]   2> 940061 INFO  (coreLoadExecutor-2125-thread-1-pr

[...truncated too long message...]

eptor.java:415)
   [junit4]   2>        at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:198)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest.teardownClass(HdfsWriteToMultipleCollectionsTest.java:73)
   [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2>        ... 53 more
   [junit4]   2> 5339258 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[3BD7CAACFABD75FE]-worker) [    
] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_3BD7CAACFABD75FE-001
   [junit4]   2> Jan 28, 2017 5:15:22 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 35 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
{rnd_b=BlockTreeOrds(blocksize=128), _version_=PostingsFormat(name=MockRandom), 
a_t=PostingsFormat(name=LuceneVarGapFixedInterval), 
a_i=BlockTreeOrds(blocksize=128), id=BlockTreeOrds(blocksize=128)}, 
docValues:{}, maxPointsInLeafNode=1411, maxMBSortInHeap=5.641185289941632, 
sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=es-SV, 
timezone=Europe/Zurich
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 
1.8.0_102 (64-bit)/cpus=4,threads=2,free=74061152,total=531628032
   [junit4]   2> NOTE: All tests run in this JVM: [TestStressUserVersions, 
TestCustomStream, TestDelegationWithHadoopAuth, InfoHandlerTest, 
TestRandomRequestDistribution, BadComponentTest, 
DistributedFacetPivotWhiteBoxTest, SolrMetricReporterTest, 
TestUseDocValuesAsStored, TestComplexPhraseQParserPlugin, JvmMetricsTest, 
TestPostingsSolrHighlighter, UnloadDistributedZkTest, TestSolrDynamicMBean, 
TestLeaderElectionZkExpiry, DeleteReplicaTest, PingRequestHandlerTest, 
PreAnalyzedFieldManagedSchemaCloudTest, DistributedExpandComponentTest, 
TestManagedResource, ForceLeaderTest, VMParamsZkACLAndCredentialsProvidersTest, 
FullHLLTest, TestConfigSetsAPIExclusivity, BasicDistributedZk2Test, 
ZkStateWriterTest, SuggestComponentTest, TestJavabinTupleStreamParser, 
LeaderFailoverAfterPartitionTest, TestExtendedDismaxParser, 
ConnectionReuseTest, DebugComponentTest, SmileWriterTest, 
XmlUpdateRequestHandlerTest, BadIndexSchemaTest, TestIndexSearcher, 
TestRawTransformer, StressHdfsTest, DistributedQueueTest, 
FullSolrCloudDistribCmdsTest, OverseerTest, TestDefaultSearchFieldResource, 
HdfsChaosMonkeyNothingIsSafeTest, SpellPossibilityIteratorTest, 
BooleanFieldTest, PrimitiveFieldTypeTest, TestNonDefinedSimilarityFactory, 
SimpleMLTQParserTest, TolerantUpdateProcessorTest, ShowFileRequestHandlerTest, 
SuggesterTest, TestUtils, TestLMJelinekMercerSimilarityFactory, 
TestPhraseSuggestions, SSLMigrationTest, PreAnalyzedFieldTest, 
OverseerCollectionConfigSetProcessorTest, TestScoreJoinQPScore, 
DocValuesMissingTest, TestDistributedMissingSort, 
SolrCoreCheckLockOnStartupTest, BlockJoinFacetSimpleTest, TestSolrCoreParser, 
ShufflingReplicaListTransformerTest, TestSort, DOMUtilTest, 
TestTolerantUpdateProcessorRandomCloud, SuggesterWFSTTest, OverseerStatusTest, 
LeaderInitiatedRecoveryOnShardRestartTest, TestRequestForwarding, 
TermVectorComponentDistributedTest, TestExportWriter, TestInitQParser, 
TestCloudPivotFacet, CircularListTest, TestHdfsUpdateLog, 
TestIndexingPerformance, TestFieldTypeResource, TestSimpleQParserPlugin, 
TestRandomCollapseQParserPlugin, LeaderElectionTest, 
IgnoreCommitOptimizeUpdateProcessorFactoryTest, ConnectionManagerTest, 
TestFieldCacheReopen, TestPKIAuthenticationPlugin, CdcrUpdateLogTest, 
TestSubQueryTransformer, FieldMutatingUpdateProcessorTest, 
BigEndianAscendingWordSerializerTest, ResponseLogComponentTest, TestLFUCache, 
HttpSolrCallGetCoreTest, TestFieldCollectionResource, 
StatelessScriptUpdateProcessorFactoryTest, SolrIndexMetricsTest, 
TestBulkSchemaAPI, ExitableDirectoryReaderTest, TestUninvertingReader, 
TestXmlQParserPlugin, CollectionsAPISolrJTest, LukeRequestHandlerTest, 
SmileWriterTest, TestWriterPerf, AssignTest, DistributedFacetPivotLongTailTest, 
TestAuthenticationFramework, TestFieldCacheSortRandom, 
DistributedIntervalFacetingTest, SolrTestCaseJ4Test, 
TestCursorMarkWithoutUniqueKey, TestHighlightDedupGrouping, 
TestSimpleTrackingShardHandler, TestTolerantSearch, 
AsyncCallRequestStatusResponseTest, CdcrRequestHandlerTest, 
CloudExitableDirectoryReaderTest, CollectionReloadTest, 
CollectionTooManyReplicasTest, ConcurrentDeleteAndCreateCollectionTest, 
DistribDocExpirationUpdateProcessorTest, DistributedVersionInfoTest, 
LeaderFailureAfterFreshStartTest, MigrateRouteKeyTest, 
MissingSegmentRecoveryTest, MultiThreadedOCPTest, RemoteQueryErrorTest, 
ReplaceNodeTest, ShardSplitTest, SharedFSAutoReplicaFailoverTest, 
SharedFSAutoReplicaFailoverUtilsTest, SolrCLIZkUtilsTest, TestCloudInspectUtil, 
TestCloudRecovery, TestClusterProperties, TestConfigSetsAPI, 
TestConfigSetsAPIZkFailure, TestDeleteCollectionOnDownNodes, 
TestDownShardTolerantSearch, TestExclusionRuleCollectionAccess, 
TestLeaderInitiatedRecoveryThread, TestLocalFSCloudBackupRestore, TestLockTree, 
TestMiniSolrCloudClusterSSL, TestOnReconnectListenerSupport, 
TestRandomFlRTGCloud, TestRebalanceLeaders, TestReplicaProperties, 
TestRequestStatusCollectionAPI, TestSegmentSorting, TestShortCircuitedRequests, 
TestSizeLimitedDistributedMap, TestSolrCloudWithSecureImpersonation, 
TestStressLiveNodes, HdfsChaosMonkeySafeLeaderTest, HdfsNNFailoverTest, 
HdfsSyncSliceTest, HdfsWriteToMultipleCollectionsTest]
   [junit4] Completed [547/681 (3!)] on J2 in 59.83s, 1 test, 1 failure <<< 
FAILURES!

[...truncated 62874 lines...]



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org