Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/677/

5 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([FEA2EACA01B8699E]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([FEA2EACA01B8699E]:0)


REGRESSION:  
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 
seconds
        at 
__randomizedtesting.SeedInfo.seed([FEA2EACA01B8699E:7F4464D276E709A2]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:178)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:137)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:132)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:834)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.createCollection(CollectionsAPIDistributedZkTest.java:1332)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:1259)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:210)
        at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


REGRESSION:  
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
Error executing query

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Error executing query
        at 
__randomizedtesting.SeedInfo.seed([FEA2EACA01B8699E:7F4464D276E709A2]:0)
        at 
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:100)
        at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
        at 
org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:223)
        at 
org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:165)
        at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:414)
        at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:144)
        at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.ParseException: Invalid content type: 
        at org.apache.http.entity.ContentType.parse(ContentType.java:233)
        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:513)
        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
        at 
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
        ... 46 more


FAILED:  
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1415292543590,generation=2,filelist=[_14q.fdt, 
_14q.fdx, _14q.fnm, _14q.nvd, _14q.nvm, _14q.si, _14q_Asserting_0.doc, 
_14q_Asserting_0.tim, _14q_Asserting_0.tip, _14r.cfe, _14r.cfs, _14r.si, 
_14s.cfe, _14s.cfs, _14s.si, _14t.cfe, _14t.cfs, _14t.si, _14u.cfe, _14u.cfs, 
_14u.si, _14v.cfe, _14v.cfs, _14v.si, _14w.cfe, _14w.cfs, _14w.si, _14x.cfe, 
_14x.cfs, _14x.si, _14y.cfe, _14y.cfs, _14y.si, _14z.cfe, _14z.cfs, _14z.si, 
_150.cfe, _150.cfs, _150.si, _151.cfe, _151.cfs, _151.si, _152.cfe, _152.cfs, 
_152.si, _153.cfe, _153.cfs, _153.si, _154.cfe, _154.cfs, _154.si, _155.cfe, 
_155.cfs, _155.si, _156.cfe, _156.cfs, _156.si, _157.cfe, _157.cfs, _157.si, 
_158.cfe, _158.cfs, _158.si, _159.cfe, _159.cfs, _159.si, _15a.cfe, _15a.cfs, 
_15a.si, _15b.cfe, _15b.cfs, _15b.si, _15c.cfe, _15c.cfs, _15c.si, _15d.cfe, 
_15d.cfs, _15d.si, _g0.fdt, _g0.fdx, _g0.fnm, _g0.nvd, _g0.nvm, _g0.si, 
_g0_Asserting_0.doc, _g0_Asserting_0.tim, _g0_Asserting_0.tip, _vd.fdt, 
_vd.fdx, _vd.fnm, _vd.nvd, _vd.nvm, _vd.si, _vd_Asserting_0.doc, 
_vd_Asserting_0.tim, _vd_Asserting_0.tip, segments_2]}]> but 
was:<[{indexVersion=1415292543590,generation=3,filelist=[_15e.fdt, _15e.fdx, 
_15e.fnm, _15e.nvd, _15e.nvm, _15e.si, _15e_Asserting_0.doc, 
_15e_Asserting_0.tim, _15e_Asserting_0.tip, _g0.fdt, _g0.fdx, _g0.fnm, _g0.nvd, 
_g0.nvm, _g0.si, _g0_Asserting_0.doc, _g0_Asserting_0.tim, _g0_Asserting_0.tip, 
_vd.fdt, _vd.fdx, _vd.fnm, _vd.nvd, _vd.nvm, _vd.si, _vd_Asserting_0.doc, 
_vd_Asserting_0.tim, _vd_Asserting_0.tip, segments_3]}, 
{indexVersion=1415292543590,generation=2,filelist=[_14q.fdt, _14q.fdx, 
_14q.fnm, _14q.nvd, _14q.nvm, _14q.si, _14q_Asserting_0.doc, 
_14q_Asserting_0.tim, _14q_Asserting_0.tip, _14r.cfe, _14r.cfs, _14r.si, 
_14s.cfe, _14s.cfs, _14s.si, _14t.cfe, _14t.cfs, _14t.si, _14u.cfe, _14u.cfs, 
_14u.si, _14v.cfe, _14v.cfs, _14v.si, _14w.cfe, _14w.cfs, _14w.si, _14x.cfe, 
_14x.cfs, _14x.si, _14y.cfe, _14y.cfs, _14y.si, _14z.cfe, _14z.cfs, _14z.si, 
_150.cfe, _150.cfs, _150.si, _151.cfe, _151.cfs, _151.si, _152.cfe, _152.cfs, 
_152.si, _153.cfe, _153.cfs, _153.si, _154.cfe, _154.cfs, _154.si, _155.cfe, 
_155.cfs, _155.si, _156.cfe, _156.cfs, _156.si, _157.cfe, _157.cfs, _157.si, 
_158.cfe, _158.cfs, _158.si, _159.cfe, _159.cfs, _159.si, _15a.cfe, _15a.cfs, 
_15a.si, _15b.cfe, _15b.cfs, _15b.si, _15c.cfe, _15c.cfs, _15c.si, _15d.cfe, 
_15d.cfs, _15d.si, _g0.fdt, _g0.fdx, _g0.fnm, _g0.nvd, _g0.nvm, _g0.si, 
_g0_Asserting_0.doc, _g0_Asserting_0.tim, _g0_Asserting_0.tip, _vd.fdt, 
_vd.fdx, _vd.fnm, _vd.nvd, _vd.nvm, _vd.si, _vd_Asserting_0.doc, 
_vd_Asserting_0.tim, _vd_Asserting_0.tip, segments_2]}]>

Stack Trace:
java.lang.AssertionError: 
expected:<[{indexVersion=1415292543590,generation=2,filelist=[_14q.fdt, 
_14q.fdx, _14q.fnm, _14q.nvd, _14q.nvm, _14q.si, _14q_Asserting_0.doc, 
_14q_Asserting_0.tim, _14q_Asserting_0.tip, _14r.cfe, _14r.cfs, _14r.si, 
_14s.cfe, _14s.cfs, _14s.si, _14t.cfe, _14t.cfs, _14t.si, _14u.cfe, _14u.cfs, 
_14u.si, _14v.cfe, _14v.cfs, _14v.si, _14w.cfe, _14w.cfs, _14w.si, _14x.cfe, 
_14x.cfs, _14x.si, _14y.cfe, _14y.cfs, _14y.si, _14z.cfe, _14z.cfs, _14z.si, 
_150.cfe, _150.cfs, _150.si, _151.cfe, _151.cfs, _151.si, _152.cfe, _152.cfs, 
_152.si, _153.cfe, _153.cfs, _153.si, _154.cfe, _154.cfs, _154.si, _155.cfe, 
_155.cfs, _155.si, _156.cfe, _156.cfs, _156.si, _157.cfe, _157.cfs, _157.si, 
_158.cfe, _158.cfs, _158.si, _159.cfe, _159.cfs, _159.si, _15a.cfe, _15a.cfs, 
_15a.si, _15b.cfe, _15b.cfs, _15b.si, _15c.cfe, _15c.cfs, _15c.si, _15d.cfe, 
_15d.cfs, _15d.si, _g0.fdt, _g0.fdx, _g0.fnm, _g0.nvd, _g0.nvm, _g0.si, 
_g0_Asserting_0.doc, _g0_Asserting_0.tim, _g0_Asserting_0.tip, _vd.fdt, 
_vd.fdx, _vd.fnm, _vd.nvd, _vd.nvm, _vd.si, _vd_Asserting_0.doc, 
_vd_Asserting_0.tim, _vd_Asserting_0.tip, segments_2]}]> but 
was:<[{indexVersion=1415292543590,generation=3,filelist=[_15e.fdt, _15e.fdx, 
_15e.fnm, _15e.nvd, _15e.nvm, _15e.si, _15e_Asserting_0.doc, 
_15e_Asserting_0.tim, _15e_Asserting_0.tip, _g0.fdt, _g0.fdx, _g0.fnm, _g0.nvd, 
_g0.nvm, _g0.si, _g0_Asserting_0.doc, _g0_Asserting_0.tim, _g0_Asserting_0.tip, 
_vd.fdt, _vd.fdx, _vd.fnm, _vd.nvd, _vd.nvm, _vd.si, _vd_Asserting_0.doc, 
_vd_Asserting_0.tim, _vd_Asserting_0.tip, segments_3]}, 
{indexVersion=1415292543590,generation=2,filelist=[_14q.fdt, _14q.fdx, 
_14q.fnm, _14q.nvd, _14q.nvm, _14q.si, _14q_Asserting_0.doc, 
_14q_Asserting_0.tim, _14q_Asserting_0.tip, _14r.cfe, _14r.cfs, _14r.si, 
_14s.cfe, _14s.cfs, _14s.si, _14t.cfe, _14t.cfs, _14t.si, _14u.cfe, _14u.cfs, 
_14u.si, _14v.cfe, _14v.cfs, _14v.si, _14w.cfe, _14w.cfs, _14w.si, _14x.cfe, 
_14x.cfs, _14x.si, _14y.cfe, _14y.cfs, _14y.si, _14z.cfe, _14z.cfs, _14z.si, 
_150.cfe, _150.cfs, _150.si, _151.cfe, _151.cfs, _151.si, _152.cfe, _152.cfs, 
_152.si, _153.cfe, _153.cfs, _153.si, _154.cfe, _154.cfs, _154.si, _155.cfe, 
_155.cfs, _155.si, _156.cfe, _156.cfs, _156.si, _157.cfe, _157.cfs, _157.si, 
_158.cfe, _158.cfs, _158.si, _159.cfe, _159.cfs, _159.si, _15a.cfe, _15a.cfs, 
_15a.si, _15b.cfe, _15b.cfs, _15b.si, _15c.cfe, _15c.cfs, _15c.si, _15d.cfe, 
_15d.cfs, _15d.si, _g0.fdt, _g0.fdx, _g0.fnm, _g0.nvd, _g0.nvm, _g0.si, 
_g0_Asserting_0.doc, _g0_Asserting_0.tim, _g0_Asserting_0.tip, _vd.fdt, 
_vd.fdx, _vd.fnm, _vd.nvd, _vd.nvm, _vd.si, _vd_Asserting_0.doc, 
_vd_Asserting_0.tim, _vd_Asserting_0.tip, segments_2]}]>
        at 
__randomizedtesting.SeedInfo.seed([FEA2EACA01B8699E:DB75F1FA71F0679D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at 
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1175)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11497 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/init-core-data-001
   [junit4]   2> 114306 T303 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl 
(true) and clientAuth (false)
   [junit4]   2> 114306 T303 oas.BaseDistributedSearchTestCase.initHostContext 
Setting hostContext system property: /
   [junit4]   2> 114314 T303 oas.SolrTestCaseJ4.setUp ###Starting 
testDistribSearch
   [junit4]   2> 114348 T303 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 114359 T304 oasc.ZkTestServer$ZKServerMain.runFromConfig 
Starting server
   [junit4]   2> 114660 T303 oasc.ZkTestServer.run start zk server on port:22662
   [junit4]   2> 114673 T303 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 114979 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 115161 T310 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@4942c49a 
name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 115161 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 115162 T303 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 115174 T303 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 115301 T303 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 115302 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 115304 T312 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@b861b68 name:ZooKeeperConnection 
Watcher:127.0.0.1:22662/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 115305 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 115305 T303 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 115313 T303 oascc.SolrZkClient.makePath makePath: 
/collections/collection1
   [junit4]   2> 115318 T303 oascc.SolrZkClient.makePath makePath: 
/collections/collection1/shards
   [junit4]   2> 115322 T303 oascc.SolrZkClient.makePath makePath: 
/collections/control_collection
   [junit4]   2> 115325 T303 oascc.SolrZkClient.makePath makePath: 
/collections/control_collection/shards
   [junit4]   2> 115330 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 115331 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/solrconfig.xml
   [junit4]   2> 115337 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 115337 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/schema.xml
   [junit4]   2> 115342 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 115343 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 115348 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 115348 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/stopwords.txt
   [junit4]   2> 115352 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 115353 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/protwords.txt
   [junit4]   2> 115357 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 115358 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/currency.xml
   [junit4]   2> 115362 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 115363 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/enumsConfig.xml
   [junit4]   2> 115367 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 115368 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/open-exchange-rates.json
   [junit4]   2> 115372 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 115373 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 115377 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 115377 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/old_synonyms.txt
   [junit4]   2> 115381 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 115382 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/synonyms.txt
   [junit4]   2> 115406 T303 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 115407 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 115409 T314 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@5ae568e3 
name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 115410 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 115410 T303 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 115414 T303 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 115415 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 115416 T316 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@624e4cd1 
name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 115417 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 115417 T303 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 115417 T303 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 115436 T303 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 115437 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 115439 T318 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@5d34f657 
name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 115439 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 115440 T303 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 115442 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf2/solrconfig.xml
   [junit4]   2> 115442 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf2/solrconfig.xml
   [junit4]   2> 115448 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf2/schema.xml
   [junit4]   2> 115449 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf2/schema.xml
   [junit4]   2> 115453 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 115454 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 115458 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf2/stopwords.txt
   [junit4]   2> 115458 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf2/stopwords.txt
   [junit4]   2> 115462 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf2/protwords.txt
   [junit4]   2> 115463 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf2/protwords.txt
   [junit4]   2> 115467 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf2/currency.xml
   [junit4]   2> 115467 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf2/currency.xml
   [junit4]   2> 115471 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 115472 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf2/open-exchange-rates.json
   [junit4]   2> 115476 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 115477 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 115480 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf2/old_synonyms.txt
   [junit4]   2> 115481 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf2/old_synonyms.txt
   [junit4]   2> 115485 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf2/synonyms.txt
   [junit4]   2> 115486 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf2/synonyms.txt
   [junit4]   2> 115490 T303 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/elevate.xml
 to /configs/conf2/elevate.xml
   [junit4]   2> 115491 T303 oascc.SolrZkClient.makePath makePath: 
/configs/conf2/elevate.xml
   [junit4]   2> 116925 T303 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 116993 T303 oejus.SslContextFactory.doStart Enabled Protocols 
[SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, 
TLSv1.1, TLSv1.2]
   [junit4]   2> 116999 T303 oejs.AbstractConnector.doStart Started 
SslSocketConnector@127.0.0.1:22668
   [junit4]   2> 117003 T303 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 117004 T303 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 117004 T303 oasc.SolrResourceLoader.locateSolrHome using 
system property solr.solr.home: 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002
   [junit4]   2> 117005 T303 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002/'
   [junit4]   2> 117040 T303 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002/solr.xml
   [junit4]   2> 117094 T303 oasc.CoreContainer.<init> New CoreContainer 
79307965
   [junit4]   2> 117094 T303 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002/]
   [junit4]   2> 117095 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
socketTimeout to: 120000
   [junit4]   2> 117096 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
urlScheme to: null
   [junit4]   2> 117096 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
connTimeout to: 15000
   [junit4]   2> 117096 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnectionsPerHost to: 20
   [junit4]   2> 117097 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnections to: 10000
   [junit4]   2> 117097 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
corePoolSize to: 0
   [junit4]   2> 117097 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maximumPoolSize to: 2147483647
   [junit4]   2> 117097 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxThreadIdleTime to: 5
   [junit4]   2> 117098 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
sizeOfQueue to: -1
   [junit4]   2> 117098 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
fairnessPolicy to: false
   [junit4]   2> 117101 T303 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 117103 T303 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 117104 T303 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 117104 T303 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 117104 T303 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:22662/solr
   [junit4]   2> 117137 T303 oasc.ZkController.checkChrootPath zkHost includes 
chroot
   [junit4]   2> 117138 T303 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 117138 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 117140 T329 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@2075ade1 
name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 117141 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 117141 T303 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 117154 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 117156 T331 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@322ba6d7 
name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 117156 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 117176 T303 oascc.SolrZkClient.makePath makePath: 
/overseer/queue
   [junit4]   2> 117191 T303 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-queue-work
   [junit4]   2> 117201 T303 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-map-running
   [junit4]   2> 117206 T303 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-map-completed
   [junit4]   2> 117211 T303 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-map-failure
   [junit4]   2> 117240 T303 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 117242 T303 oasc.ZkController.createEphemeralLiveNode Register 
node as live in ZooKeeper:/live_nodes/127.0.0.1:22668_
   [junit4]   2> 117251 T303 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:22668_
   [junit4]   2> 117255 T303 oascc.SolrZkClient.makePath makePath: 
/overseer_elect
   [junit4]   2> 117258 T303 oascc.SolrZkClient.makePath makePath: 
/overseer_elect/election
   [junit4]   2> 117261 T303 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 117285 T303 oasc.OverseerElectionContext.runLeaderProcess I am 
going to be the leader 127.0.0.1:22668_
   [junit4]   2> 117286 T303 oascc.SolrZkClient.makePath makePath: 
/overseer_elect/leader
   [junit4]   2> 117289 T303 oasc.Overseer.start Overseer 
(id=92752555564007430-127.0.0.1:22668_-n_0000000000) starting
   [junit4]   2> 117308 T303 oascc.SolrZkClient.makePath makePath: 
/overseer/queue-work
   [junit4]   2> 117345 T303 oasc.OverseerAutoReplicaFailoverThread.<init> 
Starting OverseerAutoReplicaFailoverThread 
autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=30000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 117367 T333 oasc.OverseerCollectionProcessor.run Process 
current queue of collection creations
   [junit4]   2> 117368 T303 oascc.SolrZkClient.makePath makePath: 
/clusterstate.json
   [junit4]   2> 117371 T303 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 117373 T303 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 117404 T332 oasc.Overseer$ClusterStateUpdater.run Starting to 
work on the main queue
   [junit4]   2> 117414 T303 oass.SolrDispatchFilter.init 
user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 117415 T303 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 117461 T303 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 117462 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 117464 T337 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@6ec94dc8 
name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 117465 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 117465 T303 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 117467 T303 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 117474 T303 oasc.ChaosMonkey.monkeyLog monkey: init - expire 
sessions:false cause connection loss:false
   [junit4]   2> 117475 T303 oasc.AbstractFullDistribZkTestBase.createJettys 
Creating collection1 with stateFormat=2
   [junit4]   2> 117475 T303 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 117476 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 117477 T339 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@20afaa0a 
name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 117477 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 117478 T303 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 117492 T340 oasc.DistributedQueue$LatchWatcher.process 
NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 117509 T332 oasc.Overseer$ClusterStateUpdater.buildCollection 
building a new collection: collection1
   [junit4]   2> 117510 T332 oasc.Overseer$ClusterStateUpdater.createCollection 
Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 117520 T332 oasc.Overseer$ClusterStateUpdater.createCollection 
state version collection1 2
   [junit4]   2> 117530 T332 oasc.Overseer$ClusterStateUpdater.updateZkStates 
going to create_collection /collections/collection1/state.json
   [junit4]   2> 117533 T340 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 117533 T341 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 118592 T303 oasc.AbstractFullDistribZkTestBase.createJettys 
create jetty 1
   [junit4]   2> 118593 T303 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 118599 T303 oejus.SslContextFactory.doStart Enabled Protocols 
[SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, 
TLSv1.1, TLSv1.2]
   [junit4]   2> 118602 T303 oejs.AbstractConnector.doStart Started 
SslSocketConnector@127.0.0.1:22673
   [junit4]   2> 118605 T303 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 118606 T303 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 118606 T303 oasc.SolrResourceLoader.locateSolrHome using 
system property solr.solr.home: 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-003
   [junit4]   2> 118606 T303 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-003/'
   [junit4]   2> 118642 T303 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-003/solr.xml
   [junit4]   2> 118697 T303 oasc.CoreContainer.<init> New CoreContainer 
1226200913
   [junit4]   2> 118698 T303 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-003/]
   [junit4]   2> 118699 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
socketTimeout to: 120000
   [junit4]   2> 118699 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
urlScheme to: null
   [junit4]   2> 118699 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
connTimeout to: 15000
   [junit4]   2> 118699 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnectionsPerHost to: 20
   [junit4]   2> 118700 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnections to: 10000
   [junit4]   2> 118700 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
corePoolSize to: 0
   [junit4]   2> 118700 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maximumPoolSize to: 2147483647
   [junit4]   2> 118701 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxThreadIdleTime to: 5
   [junit4]   2> 118701 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
sizeOfQueue to: -1
   [junit4]   2> 118701 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
fairnessPolicy to: false
   [junit4]   2> 118704 T303 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 118706 T303 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 118707 T303 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 118707 T303 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 118707 T303 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:22662/solr
   [junit4]   2> 118708 T303 oasc.ZkController.checkChrootPath zkHost includes 
chroot
   [junit4]   2> 118708 T303 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 118709 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 118711 T352 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@77347fda 
name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 118711 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 118712 T303 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 118714 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 118716 T354 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@41e8505a 
name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 118716 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 118727 T303 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 119737 T303 oasc.ZkController.createEphemeralLiveNode Register 
node as live in ZooKeeper:/live_nodes/127.0.0.1:22673_
   [junit4]   2> 119739 T303 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:22673_
   [junit4]   2> 119746 T303 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 119762 T303 oass.SolrDispatchFilter.init 
user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 119762 T303 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 121080 T303 oasc.AbstractFullDistribZkTestBase.createJettys 
create jetty 2
   [junit4]   2> 121086 T303 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 121093 T303 oejus.SslContextFactory.doStart Enabled Protocols 
[SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, 
TLSv1.1, TLSv1.2]
   [junit4]   2> 121095 T303 oejs.AbstractConnector.doStart Started 
SslSocketConnector@127.0.0.1:22676
   [junit4]   2> 121098 T303 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 121098 T303 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 121099 T303 oasc.SolrResourceLoader.locateSolrHome using 
system property solr.solr.home: 
../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004
   [junit4]   2> 121099 T303 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004/'
   [junit4]   2> 121130 T303 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004/solr.xml
   [junit4]   2> 121218 T303 oasc.CoreContainer.<init> New CoreContainer 
1777797839
   [junit4]   2> 121219 T303 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004/]
   [junit4]   2> 121220 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
socketTimeout to: 120000
   [junit4]   2> 121220 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
urlScheme to: null
   [junit4]   2> 121221 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
connTimeout to: 15000
   [junit4]   2> 121221 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnectionsPerHost to: 20
   [junit4]   2> 121221 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnections to: 10000
   [junit4]   2> 121221 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
corePoolSize to: 0
   [junit4]   2> 121222 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maximumPoolSize to: 2147483647
   [junit4]   2> 121222 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxThreadIdleTime to: 5
   [junit4]   2> 121222 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
sizeOfQueue to: -1
   [junit4]   2> 121223 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
fairnessPolicy to: false
   [junit4]   2> 121225 T303 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 121228 T303 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 121228 T303 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 121228 T303 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 121229 T303 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:22662/solr
   [junit4]   2> 121229 T303 oasc.ZkController.checkChrootPath zkHost includes 
chroot
   [junit4]   2> 121229 T303 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 121230 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 121233 T367 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@2e40dc59 
name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 121233 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 121234 T303 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 121237 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 121239 T369 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@54505bcd 
name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 121239 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 121251 T303 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 122261 T303 oasc.ZkController.createEphemeralLiveNode Register 
node as live in ZooKeeper:/live_nodes/127.0.0.1:22676_
   [junit4]   2> 122263 T303 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:22676_
   [junit4]   2> 122269 T303 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 122276 T303 oass.SolrDispatchFilter.init 
user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 122276 T303 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 123924 T303 oasc.AbstractFullDistribZkTestBase.createJettys 
create jetty 3
   [junit4]   2> 123925 T303 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 123934 T303 oejus.SslContextFactory.doStart Enabled Protocols 
[SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, 
TLSv1.1, TLSv1.2]
   [junit4]   2> 123938 T303 oejs.AbstractConnector.doStart Started 
SslSocketConnector@127.0.0.1:22679
   [junit4]   2> 123942 T303 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 123943 T303 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 123943 T303 oasc.SolrResourceLoader.locateSolrHome using 
system property solr.solr.home: 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005
   [junit4]   2> 123943 T303 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005/'
   [junit4]   2> 123990 T303 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005/solr.xml
   [junit4]   2> 124074 T303 oasc.CoreContainer.<init> New CoreContainer 
770659873
   [junit4]   2> 124074 T303 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005/]
   [junit4]   2> 124076 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
socketTimeout to: 120000
   [junit4]   2> 124076 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
urlScheme to: null
   [junit4]   2> 124077 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
connTimeout to: 15000
   [junit4]   2> 124077 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnectionsPerHost to: 20
   [junit4]   2> 124077 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnections to: 10000
   [junit4]   2> 124078 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
corePoolSize to: 0
   [junit4]   2> 124078 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maximumPoolSize to: 2147483647
   [junit4]   2> 124079 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxThreadIdleTime to: 5
   [junit4]   2> 124079 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
sizeOfQueue to: -1
   [junit4]   2> 124080 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
fairnessPolicy to: false
   [junit4]   2> 124083 T303 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 124087 T303 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 124087 T303 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 124088 T303 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 124088 T303 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:22662/solr
   [junit4]   2> 124088 T303 oasc.ZkController.checkChrootPath zkHost includes 
chroot
   [junit4]   2> 124089 T303 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 124090 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 124092 T382 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@42e550fd 
name:ZooKeeperConnection Watcher:127.0.0.1:22662 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 124093 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 124094 T303 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 124097 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 124098 T384 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@c6a5b72 name:ZooKeeperConnection 
Watcher:127.0.0.1:22662/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 124099 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 124110 T303 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 125120 T303 oasc.ZkController.createEphemeralLiveNode Register 
node as live in ZooKeeper:/live_nodes/127.0.0.1:22679_
   [junit4]   2> 125122 T303 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:22679_
   [junit4]   2> 125128 T303 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 125136 T303 oass.SolrDispatchFilter.init 
user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 125136 T303 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 126794 T303 oasc.AbstractFullDistribZkTestBase.createJettys 
create jetty 4
   [junit4]   2> 126796 T303 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 126803 T303 oejus.SslContextFactory.doStart Enabled Protocols 
[SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, 
TLSv1.1, TLSv1.2]
   [junit4]   2> 126807 T303 oejs.AbstractConnector.doStart Started 
SslSocketConnector@127.0.0.1:22682
   [junit4]   2> 126811 T303 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 126811 T303 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 126812 T303 oasc.SolrResourceLoader.locateSolrHome using 
system property solr.solr.home: 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006
   [junit4]   2> 126812 T303 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006/'
   [junit4]   2> 126864 T303 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006/solr.xml
   [junit4]   2> 126947 T303 oasc.CoreContainer.<init> New CoreContainer 
1469336945
   [junit4]   2> 126948 T303 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006/]
   [junit4]   2> 126949 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
socketTimeout to: 120000
   [junit4]   2> 126949 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
urlScheme to: null
   [junit4]   2> 126950 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
connTimeout to: 15000
   [junit4]   2> 126950 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnectionsPerHost to: 20
   [junit4]   2> 126951 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnections to: 10000
   [junit4]   2> 126951 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
corePoolSize to: 0
   [junit4]   2> 126951 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maximumPoolSize to: 2147483647
   [junit4]   2> 126952 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
maxThreadIdleTime to: 5
   [junit4]   2> 126952 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
sizeOfQueue to: -1
   [junit4]   2> 126953 T303 oashc.HttpShardHandlerFactory.getParameter Setting 
fairnessPolicy to: false
   [junit4]   2> 126957 T303 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 126960 T303 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 126961 T303 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 126961 T303 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 126962 T303 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:22662/solr
   [junit4]   2> 126962 T303 oasc.ZkController.checkChrootPath zkHost includes 
chroot
   [junit4]   2> 126962 T303 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 126964 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 126966 T397 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@1f4e7db name:ZooKeeperConnection 
Watcher:127.0.0.1:22662 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 126967 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 126967 T303 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 126970 T303 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 126972 T399 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@79025994 
name:ZooKeeperConnection Watcher:127.0.0.1:22662/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 126972 T303 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 126982 T303 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 127991 T303 oasc.ZkController.createEphemeralLiveNode Register 
node as live in ZooKeeper:/live_nodes/127.0.0.1:22682_
   [junit4]   2> 127993 T303 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:22682_
   [junit4]   2> 128000 T303 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 128007 T303 oass.SolrDispatchFilter.init 
user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 128008 T303 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 128323 T349 oasha.CollectionsHandler.handleCreateAction 
Creating Collection : 
name=solrj_collection&wt=javabin&collection.configName=conf1&replicationFactor=2&action=CREATE&numShards=2&version=2&router.field=myOwnField
   [junit4]   2> 128326 T340 oasc.DistributedQueue$LatchWatcher.process 
NodeChildrenChanged fired on path /overseer/collection-queue-work state 
SyncConnected
   [junit4]   2> 128328 T333 oasc.OverseerCollectionProcessor.run Overseer 
Collection Processor: Get the message 
id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>          "operation":"create",
   [junit4]   2>          "fromApi":"true",
   [junit4]   2>          "name":"solrj_collection",
   [junit4]   2>          "replicationFactor":"2",
   [junit4]   2>          "collection.configName":"conf1",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "router.field":"myOwnField"}
   [junit4]   2> 128335 T403 oasc.OverseerCollectionProcessor.processMessage 
WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2>          "operation":"create",
   [junit4]   2>          "fromApi":"true",
   [junit4]   2>          "name":"solrj_collection",
   [junit4]   2>          "replicationFactor":"2",
   [junit4]   2>          "collection.configName":"conf1",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "router.field":"myOwnField"}
   [junit4]   2> 128343 T403 oasc.OverseerCollectionProcessor.createConfNode 
creating collections conf node /collections/solrj_collection 
   [junit4]   2> 128344 T403 oascc.SolrZkClient.makePath makePath: 
/collections/solrj_collection
   [junit4]   2> 128351 T340 oasc.DistributedQueue$LatchWatcher.process 
NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 128358 T332 oasc.Overseer$ClusterStateUpdater.buildCollection 
building a new collection: solrj_collection
   [junit4]   2> 128358 T332 oasc.Overseer$ClusterStateUpdater.createCollection 
Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 128358 T332 oasc.Overseer$ClusterStateUpdater.createCollection 
state version solrj_collection 1
   [junit4]   2> 128364 T370 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128364 T340 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128364 T355 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128364 T400 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128364 T341 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128364 T385 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128451 T403 oasc.OverseerCollectionProcessor.createCollection 
Creating SolrCores for new collection solrj_collection, shardNames [shard1, 
shard2] , replicationFactor : 2
   [junit4]   2> 128452 T403 oasc.OverseerCollectionProcessor.createCollection 
Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of 
collection solrj_collection on 127.0.0.1:22682_
   [junit4]   2> 128454 T403 oasc.OverseerCollectionProcessor.createCollection 
Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of 
collection solrj_collection on 127.0.0.1:22668_
   [junit4]   2> 128457 T403 oasc.OverseerCollectionProcessor.createCollection 
Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of 
collection solrj_collection on 127.0.0.1:22679_
   [junit4]   2> 128460 T403 oasc.OverseerCollectionProcessor.createCollection 
Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of 
collection solrj_collection on 127.0.0.1:22676_
   [junit4]   2> 128602 T323 oasha.CoreAdminHandler.handleCreateAction core 
create command 
name=solrj_collection_shard1_replica2&wt=javabin&shard=shard1&collection.configName=conf1&action=CREATE&numShards=2&version=2&qt=/admin/cores&collection=solrj_collection
   [junit4]   2> 128606 T374 oasha.CoreAdminHandler.handleCreateAction core 
create command 
name=solrj_collection_shard2_replica1&wt=javabin&shard=shard2&collection.configName=conf1&action=CREATE&numShards=2&version=2&qt=/admin/cores&collection=solrj_collection
   [junit4]   2> 128610 T389 oasha.CoreAdminHandler.handleCreateAction core 
create command 
name=solrj_collection_shard1_replica1&wt=javabin&shard=shard1&collection.configName=conf1&action=CREATE&numShards=2&version=2&qt=/admin/cores&collection=solrj_collection
   [junit4]   2> 128612 T361 oasha.CoreAdminHandler.handleCreateAction core 
create command 
name=solrj_collection_shard2_replica2&wt=javabin&shard=shard2&collection.configName=conf1&action=CREATE&numShards=2&version=2&qt=/admin/cores&collection=solrj_collection
   [junit4]   2> 128615 T374 oasc.ZkController.publish publishing 
core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 128616 T323 oasc.ZkController.publish publishing 
core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 128616 T389 oasc.ZkController.publish publishing 
core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 128617 T361 oasc.ZkController.publish publishing 
core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 128625 T340 oasc.DistributedQueue$LatchWatcher.process 
NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 128625 T323 oasc.ZkController.waitForCoreNodeName look for our 
core node name
   [junit4]   2> 128626 T361 oasc.ZkController.waitForCoreNodeName look for our 
core node name
   [junit4]   2> 128625 T389 oasc.ZkController.waitForCoreNodeName look for our 
core node name
   [junit4]   2> 128626 T374 oasc.ZkController.waitForCoreNodeName look for our 
core node name
   [junit4]   2> 128638 T332 oasc.Overseer$ClusterStateUpdater.updateState 
Update state numShards=2 message={
   [junit4]   2>          "roles":null,
   [junit4]   2>          "operation":"state",
   [junit4]   2>          "base_url":"https://127.0.0.1:22682";,
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "node_name":"127.0.0.1:22682_",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "core":"solrj_collection_shard1_replica1",
   [junit4]   2>          "collection":"solrj_collection",
   [junit4]   2>          "state":"down"}
   [junit4]   2> 128651 T332 oasc.Overseer$ClusterStateUpdater.updateState 
Update state numShards=2 message={
   [junit4]   2>          "roles":null,
   [junit4]   2>          "operation":"state",
   [junit4]   2>          "base_url":"https://127.0.0.1:22668";,
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "node_name":"127.0.0.1:22668_",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "core":"solrj_collection_shard1_replica2",
   [junit4]   2>          "collection":"solrj_collection",
   [junit4]   2>          "state":"down"}
   [junit4]   2> 128656 T340 oasc.DistributedQueue$LatchWatcher.process 
NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 128658 T332 oasc.Overseer$ClusterStateUpdater.updateState 
Update state numShards=2 message={
   [junit4]   2>          "roles":null,
   [junit4]   2>          "operation":"state",
   [junit4]   2>          "base_url":"https://127.0.0.1:22679";,
   [junit4]   2>          "shard":"shard2",
   [junit4]   2>          "node_name":"127.0.0.1:22679_",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "core":"solrj_collection_shard2_replica1",
   [junit4]   2>          "collection":"solrj_collection",
   [junit4]   2>          "state":"down"}
   [junit4]   2> 128665 T332 oasc.Overseer$ClusterStateUpdater.updateState 
Update state numShards=2 message={
   [junit4]   2>          "roles":null,
   [junit4]   2>          "operation":"state",
   [junit4]   2>          "base_url":"https://127.0.0.1:22676";,
   [junit4]   2>          "shard":"shard2",
   [junit4]   2>          "node_name":"127.0.0.1:22676_",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "core":"solrj_collection_shard2_replica2",
   [junit4]   2>          "collection":"solrj_collection",
   [junit4]   2>          "state":"down"}
   [junit4]   2> 128669 T340 oasc.DistributedQueue$LatchWatcher.process 
NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 128671 T385 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128672 T400 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128672 T370 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128671 T340 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128671 T355 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 128671 T341 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 129627 T361 oasc.ZkController.waitForShardId waiting to find 
shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 129627 T323 oasc.ZkController.waitForShardId waiting to find 
shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 129627 T389 oasc.ZkController.waitForShardId waiting to find 
shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 129627 T374 oasc.ZkController.waitForShardId waiting to find 
shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 129627 T361 oasc.ZkController.createCollectionZkNode Check for 
collection zkNode:solrj_collection
   [junit4]   2> 129629 T374 oasc.ZkController.createCollectionZkNode Check for 
collection zkNode:solrj_collection
   [junit4]   2> 129628 T389 oasc.ZkController.createCollectionZkNode Check for 
collection zkNode:solrj_collection
   [junit4]   2> 129628 T323 oasc.ZkController.createCollectionZkNode Check for 
collection zkNode:solrj_collection
   [junit4]   2> 129630 T374 oasc.ZkController.createCollectionZkNode 
Collection zkNode exists
   [junit4]   2> 129631 T323 oasc.ZkController.createCollectionZkNode 
Collection zkNode exists
   [junit4]   2> 129630 T361 oasc.ZkController.createCollectionZkNode 
Collection zkNode exists
   [junit4]   2> 129631 T323 oascc.ZkStateReader.readConfigName Load collection 
config from:/collections/solrj_collection
   [junit4]   2> 129631 T374 oascc.ZkStateReader.readConfigName Load collection 
config from:/collections/solrj_collection
   [junit4]   2> 129630 T389 oasc.ZkController.createCollectionZkNode 
Collection zkNode exists
   [junit4]   2> 129632 T361 oascc.ZkStateReader.readConfigName Load collection 
config from:/collections/solrj_collection
   [junit4]   2> 129633 T323 oascc.ZkStateReader.readConfigName 
path=/collections/solrj_collection configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 129633 T389 oascc.ZkStateReader.readConfigName Load collection 
config from:/collections/solrj_collection
   [junit4]   2> 129634 T374 oascc.ZkStateReader.readConfigName 
path=/collections/solrj_collection configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 129634 T361 oascc.ZkStateReader.readConfigName 
path=/collections/solrj_collection configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 129635 T389 oascc.ZkStateReader.readConfigName 
path=/collections/solrj_collection configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 129640 T374 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005/solrj_collection_shard2_replica1/'
   [junit4]   2> 129640 T389 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006/solrj_collection_shard1_replica1/'
   [junit4]   2> 129640 T323 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002/solrj_collection_shard1_replica2/'
   [junit4]   2> 129640 T361 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004/solrj_collection_shard2_replica2/'
   [junit4]   2> 129852 T374 oasc.SolrConfig.<init> Using Lucene MatchVersion: 
6.0.0
   [junit4]   2> 129853 T323 oasc.SolrConfig.<init> Using Lucene MatchVersion: 
6.0.0
   [junit4]   2> 129854 T361 oasc.SolrConfig.<init> Using Lucene MatchVersion: 
6.0.0
   [junit4]   2> 129858 T389 oasc.SolrConfig.<init> Using Lucene MatchVersion: 
6.0.0
   [junit4]   2> 129915 T361 oasc.SolrConfig.<init> Loaded SolrConfig: 
solrconfig.xml
   [junit4]   2> 129917 T374 oasc.SolrConfig.<init> Loaded SolrConfig: 
solrconfig.xml
   [junit4]   2> 129917 T361 oass.IndexSchema.readSchema Reading Solr Schema 
from /configs/conf1/schema.xml
   [junit4]   2> 129918 T323 oasc.SolrConfig.<init> Loaded SolrConfig: 
solrconfig.xml
   [junit4]   2> 129919 T374 oass.IndexSchema.readSchema Reading Solr Schema 
from /configs/conf1/schema.xml
   [junit4]   2> 129920 T323 oass.IndexSchema.readSchema Reading Solr Schema 
from /configs/conf1/schema.xml
   [junit4]   2> 129921 T389 oasc.SolrConfig.<init> Loaded SolrConfig: 
solrconfig.xml
   [junit4]   2> 129922 T389 oass.IndexSchema.readSchema Reading Solr Schema 
from /configs/conf1/schema.xml
   [junit4]   2> 129931 T361 oass.IndexSchema.readSchema 
[solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 129948 T374 oass.IndexSchema.readSchema 
[solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 129950 T323 oass.IndexSchema.readSchema 
[solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 129954 T389 oass.IndexSchema.readSchema 
[solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 130439 T374 oass.OpenExchangeRatesOrgProvider.init Initialized 
with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 130443 T323 oass.OpenExchangeRatesOrgProvider.init Initialized 
with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 130443 T389 oass.OpenExchangeRatesOrgProvider.init Initialized 
with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 130444 T361 oass.OpenExchangeRatesOrgProvider.init Initialized 
with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 130453 T374 oass.IndexSchema.readSchema default search field 
in schema is text
   [junit4]   2> 130453 T361 oass.IndexSchema.readSchema default search field 
in schema is text
   [junit4]   2> 130454 T389 oass.IndexSchema.readSchema default search field 
in schema is text
   [junit4]   2> 130455 T323 oass.IndexSchema.readSchema default search field 
in schema is text
   [junit4]   2> 130456 T374 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 130457 T361 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 130457 T389 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 130458 T323 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 130481 T389 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 130482 T323 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 130482 T374 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 130483 T361 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 130488 T323 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 130488 T374 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 130489 T389 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 130490 T361 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 130494 T323 oass.OpenExchangeRatesOrgProvider.reload Reloading 
exchange rates from open-exchange-rates.json
   [junit4]   2> 130495 T374 oass.OpenExchangeRatesOrgProvider.reload Reloading 
exchange rates from open-exchange-rates.json
   [junit4]   2> 130495 T323 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 130496 T361 oass.OpenExchangeRatesOrgProvider.reload Reloading 
exchange rates from open-exchange-rates.json
   [junit4]   2> 130496 T389 oass.OpenExchangeRatesOrgProvider.reload Reloading 
exchange rates from open-exchange-rates.json
   [junit4]   2> 130497 T374 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 130498 T361 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 130499 T389 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 130497 T323 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 130499 T389 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 130499 T361 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 130498 T374 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 130501 T361 oass.OpenExchangeRatesOrgProvider.reload Reloading 
exchange rates from open-exchange-rates.json
   [junit4]   2> 130500 T389 oass.OpenExchangeRatesOrgProvider.reload Reloading 
exchange rates from open-exchange-rates.json
   [junit4]   2> 130500 T323 oass.OpenExchangeRatesOrgProvider.reload Reloading 
exchange rates from open-exchange-rates.json
   [junit4]   2> 130501 T374 oass.OpenExchangeRatesOrgProvider.reload Reloading 
exchange rates from open-exchange-rates.json
   [junit4]   2> 130502 T361 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 130503 T361 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 130503 T389 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 130504 T361 oasc.CoreContainer.create Creating SolrCore 
'solrj_collection_shard2_replica2' using configuration from collection 
solrj_collection
   [junit4]   2> 130504 T374 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 130503 T323 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 130505 T374 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 130504 T361 oasc.SolrCore.initDirectoryFactory 
org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 130504 T389 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 130506 T361 oasc.SolrCore.<init> 
[solrj_collection_shard2_replica2] Opening new SolrCore at 
../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004/solrj_collection_shard2_replica2/,
 
dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-004/solrj_collection_shard2_replica2/data/
   [junit4]   2> 130506 T374 oasc.CoreContainer.create Creating SolrCore 
'solrj_collection_shard2_replica1' using configuration from collection 
solrj_collection
   [junit4]   2> 130505 T323 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 130508 T374 oasc.SolrCore.initDirectoryFactory 
org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 130508 T361 oasc.JmxMonitoredMap.<init> JMX monitoring is 
enabled. Adding Solr mbeans to JMX Server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1ac9a702
   [junit4]   2> 130507 T389 oasc.CoreContainer.create Creating SolrCore 
'solrj_collection_shard1_replica1' using configuration from collection 
solrj_collection
   [junit4]   2> 130509 T374 oasc.SolrCore.<init> 
[solrj_collection_shard2_replica1] Opening new SolrCore at 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005/solrj_collection_shard2_replica1/,
 
dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-005/solrj_collection_shard2_replica1/data/
   [junit4]   2> 130508 T323 oasc.CoreContainer.create Creating SolrCore 
'solrj_collection_shard1_replica2' using configuration from collection 
solrj_collection
   [junit4]   2> 130510 T374 oasc.JmxMonitoredMap.<init> JMX monitoring is 
enabled. Adding Solr mbeans to JMX Server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1ac9a702
   [junit4]   2> 130510 T389 oasc.SolrCore.initDirectoryFactory 
org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 130511 T323 oasc.SolrCore.initDirectoryFactory 
org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 130512 T389 oasc.SolrCore.<init> 
[solrj_collection_shard1_replica1] Opening new SolrCore at 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006/solrj_collection_shard1_replica1/,
 
dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-006/solrj_collection_shard1_replica1/data/
   [junit4]   2> 130512 T323 oasc.SolrCore.<init> 
[solrj_collection_shard1_replica2] Opening new SolrCore at 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002/solrj_collection_shard1_replica2/,
 
dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-FEA2EACA01B8699E-001/tempDir-002/solrj_collection_shard1_replica2/data/
   [junit4]   2> 130513 T389 oasc.JmxMonitoredMap.<init> JMX monitoring is 
enabled. Adding Solr mbeans to JMX Server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@1ac9a702
   [junit4]   2> 130513 T323 oasc.JmxMonitoredMap.<init> JMX monitoring is 
enabled. Adding Solr mbeans to JMX Server:

[...truncated too long message...]

junit4]   2>            at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>          65) Thread[id=2186, 
name=searcherExecutor-1123-thread-1, state=WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2>                at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>                at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2>                at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2>                at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>                at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2>                at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2>                at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2>                at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>          66) Thread[id=2172, name=qtp1257495698-2172 Acceptor0 
SocketConnector@127.0.0.1:31806, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2>                at java.net.PlainSocketImpl.socketAccept(Native 
Method)
   [junit4]   2>                at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4]   2>                at 
java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4]   2>                at 
java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4]   2>                at 
org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
   [junit4]   2>                at 
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2>                at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2>                at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2>                at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>          67) Thread[id=1873, 
name=coreZkRegister-1018-thread-1, state=WAITING, 
group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2>                at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>                at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2>                at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2>                at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>                at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2>                at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2>                at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2>                at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>          68) Thread[id=1934, name=qtp1892819812-1934 Acceptor0 
SocketConnector@127.0.0.1:51454, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2>                at java.net.PlainSocketImpl.socketAccept(Native 
Method)
   [junit4]   2>                at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4]   2>                at 
java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4]   2>                at 
java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4]   2>                at 
org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
   [junit4]   2>                at 
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2>                at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2>                at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2>                at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>          69) Thread[id=1875, name=qtp1669094124-1875 Acceptor0 
SocketConnector@127.0.0.1:27878, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2>                at java.net.PlainSocketImpl.socketAccept(Native 
Method)
   [junit4]   2>                at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4]   2>                at 
java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4]   2>                at 
java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4]   2>                at 
org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
   [junit4]   2>                at 
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2>                at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2>                at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2>                at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>          70) Thread[id=1913, name=qtp1383421744-1913 Acceptor0 
SocketConnector@127.0.0.1:30451, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2>                at java.net.PlainSocketImpl.socketAccept(Native 
Method)
   [junit4]   2>                at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4]   2>                at 
java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4]   2>                at 
java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4]   2>                at 
org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
   [junit4]   2>                at 
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2>                at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2>                at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2>                at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50), 
sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=lv_LV, 
timezone=US/Michigan
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 
(64-bit)/cpus=16,threads=72,free=123046616,total=225968128
   [junit4]   2> NOTE: All tests run in this JVM: [TestExceedMaxTermLength, 
TestSchemaResource, TestSolr4Spatial2, PathHierarchyTokenizerFactoryTest, 
TestAtomicUpdateErrorCases, TestBinaryField, TestLMDirichletSimilarityFactory, 
DateFieldTest, RequiredFieldsTest, AutoCommitTest, DebugComponentTest, 
TestStandardQParsers, DeleteReplicaTest, TestDefaultSearchFieldResource, 
TestSweetSpotSimilarityFactory, SolrRequestParserTest, 
ShowFileRequestHandlerTest, SystemInfoHandlerTest, TestQuerySenderNoQuery, 
DirectUpdateHandlerTest, TestSchemaSimilarityResource, ZkControllerTest, 
FastVectorHighlighterTest, TestCodecSupport, CachingDirectoryFactoryTest, 
HdfsBasicDistributedZkTest, SolrCoreCheckLockOnStartupTest, TestCollationField, 
TestSolrConfigHandler, SuggestComponentTest, TestDistribDocBasedVersion, 
TriLevelCompositeIdRoutingTest, InfoHandlerTest, TestRecovery, 
TestManagedStopFilterFactory, TestCollectionAPI, 
TestFieldTypeCollectionResource, TestDocumentBuilder, ClusterStateTest, 
CurrencyFieldOpenExchangeTest, BlockCacheTest, TestFreeTextSuggestions, 
ConnectionManagerTest, SolrCoreTest, NoCacheHeaderTest, PreAnalyzedFieldTest, 
TestInitParams, DocValuesMissingTest, TestLFUCache, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=FEA2EACA01B8699E 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt 
-Dtests.locale=lv_LV -Dtests.timezone=US/Michigan -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 
7200000 msec).
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([FEA2EACA01B8699E]:0)
   [junit4] Completed on J1 in 7224.57s, 1 test, 2 errors <<< FAILURES!

[...truncated 25 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:532:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:473:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:502:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1358:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:965:
 There were test failures: 447 suites, 1820 tests, 1 suite-level error, 2 
errors, 2 failures, 75 ignored (9 assumptions)

Total time: 273 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #675
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 841793481 bytes
Compression is 0.0%
Took 2 min 33 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to