Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/651/

4 tests failed.
REGRESSION:  
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Could not fully create collection: acollectionafterbaddelete

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not 
fully create collection: acollectionafterbaddelete
        at 
__randomizedtesting.SeedInfo.seed([5E63E98220B52629:DF85679A57EA4615]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:569)
        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:932)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:203)
        at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
ERROR: SolrZkClient opens=19 closes=18

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=19 closes=18
        at __randomizedtesting.SeedInfo.seed([5E63E98220B52629]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:455)
        at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:188)
        at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
5 threads leaked from SUITE scope at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest:     1) Thread[id=4372, 
name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5E63E98220B52629]-SendThread(127.0.0.1:14310),
 state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)         at 
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)         at 
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)    2) 
Thread[id=4373, 
name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5E63E98220B52629]-EventThread,
 state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    
3) Thread[id=5030, name=zkCallback-353-thread-4, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=4376, 
name=zkCallback-353-thread-1, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)    5) Thread[id=4802, 
name=zkCallback-353-thread-3, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE 
scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 
   1) Thread[id=4372, 
name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5E63E98220B52629]-SendThread(127.0.0.1:14310),
 state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   2) Thread[id=4373, 
name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5E63E98220B52629]-EventThread,
 state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   3) Thread[id=5030, name=zkCallback-353-thread-4, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=4376, name=zkCallback-353-thread-1, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=4802, name=zkCallback-353-thread-3, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([5E63E98220B52629]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=5030, name=zkCallback-353-thread-4, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=4372, 
name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5E63E98220B52629]-SendThread(127.0.0.1:14310),
 state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.nio.ch.KQueueArrayWrapper.interrupt(Native Method)         at 
sun.nio.ch.KQueueArrayWrapper.interrupt(KQueueArrayWrapper.java:205)         at 
sun.nio.ch.KQueueSelectorImpl.wakeup(KQueueSelectorImpl.java:243)         at 
java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)   
      at 
java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)         
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:102)         
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)    3) 
Thread[id=4376, name=zkCallback-353-thread-1, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=4802, 
name=zkCallback-353-thread-3, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=5030, name=zkCallback-353-thread-4, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=4372, 
name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5E63E98220B52629]-SendThread(127.0.0.1:14310),
 state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.nio.ch.KQueueArrayWrapper.interrupt(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.interrupt(KQueueArrayWrapper.java:205)
        at sun.nio.ch.KQueueSelectorImpl.wakeup(KQueueSelectorImpl.java:243)
        at 
java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
        at 
java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:102)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   3) Thread[id=4376, name=zkCallback-353-thread-1, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=4802, name=zkCallback-353-thread-3, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([5E63E98220B52629]:0)




Build Log:
[...truncated 12213 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/init-core-data-001
   [junit4]   2> 1103962 T4106 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl 
(true) and clientAuth (true)
   [junit4]   2> 1103963 T4106 
oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system 
property: /
   [junit4]   2> 1103968 T4106 oas.SolrTestCaseJ4.setUp ###Starting 
testDistribSearch
   [junit4]   2> 1103969 T4106 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1103970 T4107 oasc.ZkTestServer$ZKServerMain.runFromConfig 
Starting server
   [junit4]   2> 1104070 T4106 oasc.ZkTestServer.run start zk server on 
port:14310
   [junit4]   2> 1104071 T4106 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1104072 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1104076 T4113 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@2eb314e name:ZooKeeperConnection 
Watcher:127.0.0.1:14310 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 1104076 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1104076 T4106 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 1104077 T4106 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1104079 T4106 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1104080 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1104081 T4115 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@3dd43860 
name:ZooKeeperConnection Watcher:127.0.0.1:14310/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1104082 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1104082 T4106 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 1104082 T4106 oascc.SolrZkClient.makePath makePath: 
/collections/collection1
   [junit4]   2> 1104084 T4106 oascc.SolrZkClient.makePath makePath: 
/collections/collection1/shards
   [junit4]   2> 1104086 T4106 oascc.SolrZkClient.makePath makePath: 
/collections/control_collection
   [junit4]   2> 1104088 T4106 oascc.SolrZkClient.makePath makePath: 
/collections/control_collection/shards
   [junit4]   2> 1104090 T4106 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 1104090 T4106 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/solrconfig.xml
   [junit4]   2> 1104093 T4106 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 1104094 T4106 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/schema.xml
   [junit4]   2> 1104197 T4106 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1104198 T4106 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1104200 T4106 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 1104201 T4106 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/stopwords.txt
   [junit4]   2> 1104203 T4106 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 1104204 T4106 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/protwords.txt
   [junit4]   2> 1104206 T4106 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 1104207 T4106 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/currency.xml
   [junit4]   2> 1104209 T4106 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1104210 T4106 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/enumsConfig.xml
   [junit4]   2> 1104212 T4106 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1104213 T4106 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/open-exchange-rates.json
   [junit4]   2> 1104215 T4106 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1104215 T4106 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1104218 T4106 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1104218 T4106 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/old_synonyms.txt
   [junit4]   2> 1104220 T4106 oasc.AbstractZkTestCase.putConfig put 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 1104221 T4106 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/synonyms.txt
   [junit4]   2> 1104223 T4106 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1104224 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1104225 T4117 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@266487a1 
name:ZooKeeperConnection Watcher:127.0.0.1:14310/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1104226 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1104226 T4106 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 1105648 T4106 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1105652 T4106 oejus.SslContextFactory.doStart Enabled 
Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, 
TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1105656 T4106 oejs.AbstractConnector.doStart Started 
SslSocketConnector@127.0.0.1:14317
   [junit4]   2> 1105659 T4106 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 1105659 T4106 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 1105659 T4106 oasc.SolrResourceLoader.locateSolrHome using 
system property solr.solr.home: 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-002
   [junit4]   2> 1105659 T4106 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-002/'
   [junit4]   2> 1105690 T4106 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-002/solr.xml
   [junit4]   2> 1105745 T4106 oasc.CoreContainer.<init> New CoreContainer 
648369356
   [junit4]   2> 1105745 T4106 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-002/]
   [junit4]   2> 1105746 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting socketTimeout to: 120000
   [junit4]   2> 1105747 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting urlScheme to: null
   [junit4]   2> 1105747 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting connTimeout to: 15000
   [junit4]   2> 1105747 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1105747 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxConnections to: 10000
   [junit4]   2> 1105747 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting corePoolSize to: 0
   [junit4]   2> 1105748 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1105748 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxThreadIdleTime to: 5
   [junit4]   2> 1105748 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting sizeOfQueue to: -1
   [junit4]   2> 1105748 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting fairnessPolicy to: false
   [junit4]   2> 1105750 T4106 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1105752 T4106 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1105752 T4106 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1105752 T4106 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1105753 T4106 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:14310/solr
   [junit4]   2> 1105753 T4106 oasc.ZkController.checkChrootPath zkHost 
includes chroot
   [junit4]   2> 1105753 T4106 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1105754 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1105755 T4128 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@4f35bb56 
name:ZooKeeperConnection Watcher:127.0.0.1:14310 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1105756 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1105756 T4106 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 1105757 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1105758 T4130 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@71382489 
name:ZooKeeperConnection Watcher:127.0.0.1:14310/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1105758 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1105759 T4106 oascc.SolrZkClient.makePath makePath: 
/overseer/queue
   [junit4]   2> 1105761 T4106 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-queue-work
   [junit4]   2> 1105763 T4106 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-map-running
   [junit4]   2> 1105765 T4106 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-map-completed
   [junit4]   2> 1105767 T4106 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-map-failure
   [junit4]   2> 1105769 T4106 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1105771 T4106 oasc.ZkController.createEphemeralLiveNode 
Register node as live in ZooKeeper:/live_nodes/127.0.0.1:14317_
   [junit4]   2> 1105771 T4106 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:14317_
   [junit4]   2> 1105773 T4106 oascc.SolrZkClient.makePath makePath: 
/overseer_elect
   [junit4]   2> 1105774 T4106 oascc.SolrZkClient.makePath makePath: 
/overseer_elect/election
   [junit4]   2> 1105775 T4106 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1105777 T4106 oasc.OverseerElectionContext.runLeaderProcess I 
am going to be the leader 127.0.0.1:14317_
   [junit4]   2> 1105777 T4106 oascc.SolrZkClient.makePath makePath: 
/overseer_elect/leader
   [junit4]   2> 1105778 T4106 oasc.Overseer.start Overseer 
(id=92641770658660356-127.0.0.1:14317_-n_0000000000) starting
   [junit4]   2> 1105780 T4106 oascc.SolrZkClient.makePath makePath: 
/overseer/queue-work
   [junit4]   2> 1105786 T4106 oasc.OverseerAutoReplicaFailoverThread.<init> 
Starting OverseerAutoReplicaFailoverThread 
autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=30000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1105787 T4132 oasc.OverseerCollectionProcessor.run Process 
current queue of collection creations
   [junit4]   2> 1105788 T4106 oascc.SolrZkClient.makePath makePath: 
/clusterstate.json
   [junit4]   2> 1105789 T4106 oascc.SolrZkClient.makePath makePath: 
/aliases.json
   [junit4]   2> 1105790 T4106 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 1105793 T4131 oasc.Overseer$ClusterStateUpdater.run Starting 
to work on the main queue
   [junit4]   2> 1105796 T4106 oass.SolrDispatchFilter.init 
user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1105796 T4106 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 1105799 T4106 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1105800 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1105801 T4136 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@52f0adbf 
name:ZooKeeperConnection Watcher:127.0.0.1:14310/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1105801 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1105801 T4106 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 1105802 T4106 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 1105804 T4106 oasc.ChaosMonkey.monkeyLog monkey: init - expire 
sessions:false cause connection loss:false
   [junit4]   2> 1105804 T4106 oasc.AbstractFullDistribZkTestBase.createJettys 
Creating collection1 with stateFormat=2
   [junit4]   2> 1105804 T4106 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1105805 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1105805 T4138 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@43495829 
name:ZooKeeperConnection Watcher:127.0.0.1:14310/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1105806 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1105806 T4106 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 1105807 T4139 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 1105809 T4131 
oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: 
collection1
   [junit4]   2> 1105809 T4131 
oasc.Overseer$ClusterStateUpdater.createCollection Create collection 
collection1 with shards [shard1, shard2]
   [junit4]   2> 1105809 T4131 
oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 1105811 T4131 oasc.Overseer$ClusterStateUpdater.updateZkStates 
going to create_collection /collections/collection1/state.json
   [junit4]   2> 1105812 T4139 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1105812 T4140 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1106705 T4106 oasc.AbstractFullDistribZkTestBase.createJettys 
create jetty 1
   [junit4]   2> 1106705 T4106 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1106709 T4106 oejus.SslContextFactory.doStart Enabled 
Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, 
TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1106710 T4106 oejs.AbstractConnector.doStart Started 
SslSocketConnector@127.0.0.1:14322
   [junit4]   2> 1106713 T4106 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 1106713 T4106 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 1106713 T4106 oasc.SolrResourceLoader.locateSolrHome using 
system property solr.solr.home: 
../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-003
   [junit4]   2> 1106713 T4106 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-003/'
   [junit4]   2> 1106745 T4106 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-003/solr.xml
   [junit4]   2> 1106797 T4106 oasc.CoreContainer.<init> New CoreContainer 
1516369397
   [junit4]   2> 1106797 T4106 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-003/]
   [junit4]   2> 1106798 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting socketTimeout to: 120000
   [junit4]   2> 1106798 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting urlScheme to: null
   [junit4]   2> 1106799 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting connTimeout to: 15000
   [junit4]   2> 1106799 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1106799 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxConnections to: 10000
   [junit4]   2> 1106799 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting corePoolSize to: 0
   [junit4]   2> 1106799 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1106800 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxThreadIdleTime to: 5
   [junit4]   2> 1106800 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting sizeOfQueue to: -1
   [junit4]   2> 1106800 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting fairnessPolicy to: false
   [junit4]   2> 1106802 T4106 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1106804 T4106 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1106804 T4106 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1106804 T4106 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1106804 T4106 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:14310/solr
   [junit4]   2> 1106805 T4106 oasc.ZkController.checkChrootPath zkHost 
includes chroot
   [junit4]   2> 1106805 T4106 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1106806 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1106807 T4151 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@449d2f28 
name:ZooKeeperConnection Watcher:127.0.0.1:14310 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1106808 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1106808 T4106 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 1106809 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1106810 T4153 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@325fa6e4 
name:ZooKeeperConnection Watcher:127.0.0.1:14310/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1106810 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1106814 T4106 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 1107818 T4106 oasc.ZkController.createEphemeralLiveNode 
Register node as live in ZooKeeper:/live_nodes/127.0.0.1:14322_
   [junit4]   2> 1107819 T4106 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:14322_
   [junit4]   2> 1107823 T4106 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1107828 T4106 oass.SolrDispatchFilter.init 
user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1107828 T4106 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 1109426 T4106 oasc.AbstractFullDistribZkTestBase.createJettys 
create jetty 2
   [junit4]   2> 1109427 T4106 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1109431 T4106 oejus.SslContextFactory.doStart Enabled 
Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, 
TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1109433 T4106 oejs.AbstractConnector.doStart Started 
SslSocketConnector@127.0.0.1:14328
   [junit4]   2> 1109435 T4106 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 1109435 T4106 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 1109435 T4106 oasc.SolrResourceLoader.locateSolrHome using 
system property solr.solr.home: 
../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-004
   [junit4]   2> 1109435 T4106 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-004/'
   [junit4]   2> 1109471 T4106 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-004/solr.xml
   [junit4]   2> 1109537 T4106 oasc.CoreContainer.<init> New CoreContainer 
1737484289
   [junit4]   2> 1109537 T4106 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-004/]
   [junit4]   2> 1109538 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting socketTimeout to: 120000
   [junit4]   2> 1109538 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting urlScheme to: null
   [junit4]   2> 1109538 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting connTimeout to: 15000
   [junit4]   2> 1109539 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1109539 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxConnections to: 10000
   [junit4]   2> 1109539 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting corePoolSize to: 0
   [junit4]   2> 1109539 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1109540 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxThreadIdleTime to: 5
   [junit4]   2> 1109540 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting sizeOfQueue to: -1
   [junit4]   2> 1109540 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting fairnessPolicy to: false
   [junit4]   2> 1109542 T4106 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1109544 T4106 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1109544 T4106 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1109544 T4106 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1109544 T4106 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:14310/solr
   [junit4]   2> 1109545 T4106 oasc.ZkController.checkChrootPath zkHost 
includes chroot
   [junit4]   2> 1109545 T4106 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1109546 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1109547 T4166 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@594a3452 
name:ZooKeeperConnection Watcher:127.0.0.1:14310 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1109548 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1109548 T4106 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 1109550 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1109551 T4168 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@70843e74 
name:ZooKeeperConnection Watcher:127.0.0.1:14310/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1109552 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1109557 T4106 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 1110561 T4106 oasc.ZkController.createEphemeralLiveNode 
Register node as live in ZooKeeper:/live_nodes/127.0.0.1:14328_
   [junit4]   2> 1110562 T4106 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:14328_
   [junit4]   2> 1110567 T4106 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1110574 T4106 oass.SolrDispatchFilter.init 
user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1110574 T4106 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 1112051 T4106 oasc.AbstractFullDistribZkTestBase.createJettys 
create jetty 3
   [junit4]   2> 1112052 T4106 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1112056 T4106 oejus.SslContextFactory.doStart Enabled 
Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, 
TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1112058 T4106 oejs.AbstractConnector.doStart Started 
SslSocketConnector@127.0.0.1:14336
   [junit4]   2> 1112060 T4106 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 1112060 T4106 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 1112061 T4106 oasc.SolrResourceLoader.locateSolrHome using 
system property solr.solr.home: 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-005
   [junit4]   2> 1112061 T4106 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-005/'
   [junit4]   2> 1112092 T4106 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-005/solr.xml
   [junit4]   2> 1112181 T4106 oasc.CoreContainer.<init> New CoreContainer 
1627642431
   [junit4]   2> 1112181 T4106 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-005/]
   [junit4]   2> 1112182 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting socketTimeout to: 120000
   [junit4]   2> 1112183 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting urlScheme to: null
   [junit4]   2> 1112183 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting connTimeout to: 15000
   [junit4]   2> 1112183 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1112183 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxConnections to: 10000
   [junit4]   2> 1112183 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting corePoolSize to: 0
   [junit4]   2> 1112184 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1112184 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxThreadIdleTime to: 5
   [junit4]   2> 1112184 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting sizeOfQueue to: -1
   [junit4]   2> 1112184 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting fairnessPolicy to: false
   [junit4]   2> 1112186 T4106 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1112188 T4106 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1112188 T4106 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1112189 T4106 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1112189 T4106 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:14310/solr
   [junit4]   2> 1112189 T4106 oasc.ZkController.checkChrootPath zkHost 
includes chroot
   [junit4]   2> 1112189 T4106 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1112190 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1112191 T4181 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@3ab6977f 
name:ZooKeeperConnection Watcher:127.0.0.1:14310 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1112192 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1112192 T4106 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 1112194 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1112195 T4183 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@3093158 name:ZooKeeperConnection 
Watcher:127.0.0.1:14310/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 1112195 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1112201 T4106 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 1113205 T4106 oasc.ZkController.createEphemeralLiveNode 
Register node as live in ZooKeeper:/live_nodes/127.0.0.1:14336_
   [junit4]   2> 1113206 T4106 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:14336_
   [junit4]   2> 1113209 T4106 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1113215 T4106 oass.SolrDispatchFilter.init 
user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1113215 T4106 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 1114833 T4106 oasc.AbstractFullDistribZkTestBase.createJettys 
create jetty 4
   [junit4]   2> 1114834 T4106 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1114839 T4106 oejus.SslContextFactory.doStart Enabled 
Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, 
TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1114840 T4106 oejs.AbstractConnector.doStart Started 
SslSocketConnector@127.0.0.1:14344
   [junit4]   2> 1114843 T4106 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 1114844 T4106 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 1114844 T4106 oasc.SolrResourceLoader.locateSolrHome using 
system property solr.solr.home: 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-006
   [junit4]   2> 1114845 T4106 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-006/'
   [junit4]   2> 1114890 T4106 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-006/solr.xml
   [junit4]   2> 1114969 T4106 oasc.CoreContainer.<init> New CoreContainer 
2041026899
   [junit4]   2> 1114969 T4106 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-006/]
   [junit4]   2> 1114971 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting socketTimeout to: 120000
   [junit4]   2> 1114971 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting urlScheme to: null
   [junit4]   2> 1114971 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting connTimeout to: 15000
   [junit4]   2> 1114972 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1114972 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxConnections to: 10000
   [junit4]   2> 1114972 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting corePoolSize to: 0
   [junit4]   2> 1114973 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1114973 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting maxThreadIdleTime to: 5
   [junit4]   2> 1114973 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting sizeOfQueue to: -1
   [junit4]   2> 1114974 T4106 oashc.HttpShardHandlerFactory.getParameter 
Setting fairnessPolicy to: false
   [junit4]   2> 1114976 T4106 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1114978 T4106 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1114979 T4106 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1114979 T4106 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1114980 T4106 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:14310/solr
   [junit4]   2> 1114980 T4106 oasc.ZkController.checkChrootPath zkHost 
includes chroot
   [junit4]   2> 1114980 T4106 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1114981 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1114983 T4196 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@3b697b2c 
name:ZooKeeperConnection Watcher:127.0.0.1:14310 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1114983 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1114984 T4106 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 1114986 T4106 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1114987 T4198 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@45580f75 
name:ZooKeeperConnection Watcher:127.0.0.1:14310/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1114987 T4106 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1114992 T4106 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 1115998 T4106 oasc.ZkController.createEphemeralLiveNode 
Register node as live in ZooKeeper:/live_nodes/127.0.0.1:14344_
   [junit4]   2> 1115999 T4106 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:14344_
   [junit4]   2> 1116004 T4106 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1116009 T4106 oass.SolrDispatchFilter.init 
user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1116009 T4106 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 1116157 T4143 oasha.CollectionsHandler.handleCreateAction 
Creating Collection : 
action=CREATE&router.field=myOwnField&numShards=2&version=2&replicationFactor=2&wt=javabin&name=solrj_collection&collection.configName=conf1
   [junit4]   2> 1116158 T4139 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/collection-queue-work state: 
SyncConnected type NodeChildrenChanged
   [junit4]   2> 1116160 T4132 oasc.OverseerCollectionProcessor.run Overseer 
Collection Processor: Get the message 
id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>          "operation":"create",
   [junit4]   2>          "fromApi":"true",
   [junit4]   2>          "name":"solrj_collection",
   [junit4]   2>          "replicationFactor":"2",
   [junit4]   2>          "collection.configName":"conf1",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "router.field":"myOwnField"}
   [junit4]   2> 1116161 T4202 oasc.OverseerCollectionProcessor.processMessage 
WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2>          "operation":"create",
   [junit4]   2>          "fromApi":"true",
   [junit4]   2>          "name":"solrj_collection",
   [junit4]   2>          "replicationFactor":"2",
   [junit4]   2>          "collection.configName":"conf1",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "router.field":"myOwnField"}
   [junit4]   2> 1116162 T4202 oasc.OverseerCollectionProcessor.createConfNode 
creating collections conf node /collections/solrj_collection 
   [junit4]   2> 1116163 T4202 oascc.SolrZkClient.makePath makePath: 
/collections/solrj_collection
   [junit4]   2> 1116165 T4139 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 1116168 T4131 
oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: 
solrj_collection
   [junit4]   2> 1116168 T4131 
oasc.Overseer$ClusterStateUpdater.createCollection Create collection 
solrj_collection with shards [shard1, shard2]
   [junit4]   2> 1116169 T4131 
oasc.Overseer$ClusterStateUpdater.createCollection state version 
solrj_collection 1
   [junit4]   2> 1116171 T4169 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1116171 T4154 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1116171 T4140 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1116171 T4199 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1116171 T4139 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1116171 T4184 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1116265 T4202 
oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new 
collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 1116266 T4202 
oasc.OverseerCollectionProcessor.createCollection Creating shard 
solrj_collection_shard1_replica1 as part of slice shard1 of collection 
solrj_collection on 127.0.0.1:14336_
   [junit4]   2> 1116267 T4202 
oasc.OverseerCollectionProcessor.createCollection Creating shard 
solrj_collection_shard1_replica2 as part of slice shard1 of collection 
solrj_collection on 127.0.0.1:14322_
   [junit4]   2> 1116268 T4202 
oasc.OverseerCollectionProcessor.createCollection Creating shard 
solrj_collection_shard2_replica1 as part of slice shard2 of collection 
solrj_collection on 127.0.0.1:14317_
   [junit4]   2> 1116272 T4202 
oasc.OverseerCollectionProcessor.createCollection Creating shard 
solrj_collection_shard2_replica2 as part of slice shard2 of collection 
solrj_collection on 127.0.0.1:14344_
   [junit4]   2> 1116398 T4173 oasha.CoreAdminHandler.handleCreateAction core 
create command 
shard=shard1&action=CREATE&numShards=2&version=2&wt=javabin&collection=solrj_collection&name=solrj_collection_shard1_replica1&collection.configName=conf1&qt=/admin/cores
   [junit4]   2> 1116399 T4147 oasha.CoreAdminHandler.handleCreateAction core 
create command 
shard=shard1&action=CREATE&numShards=2&version=2&wt=javabin&collection=solrj_collection&name=solrj_collection_shard1_replica2&collection.configName=conf1&qt=/admin/cores
   [junit4]   2> 1116401 T4122 oasha.CoreAdminHandler.handleCreateAction core 
create command 
shard=shard2&action=CREATE&numShards=2&version=2&wt=javabin&collection=solrj_collection&name=solrj_collection_shard2_replica1&collection.configName=conf1&qt=/admin/cores
   [junit4]   2> 1116402 T4173 oasc.ZkController.publish publishing 
core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 1116403 T4188 oasha.CoreAdminHandler.handleCreateAction core 
create command 
shard=shard2&action=CREATE&numShards=2&version=2&wt=javabin&collection=solrj_collection&name=solrj_collection_shard2_replica2&collection.configName=conf1&qt=/admin/cores
   [junit4]   2> 1116404 T4173 oasc.ZkController.waitForCoreNodeName look for 
our core node name
   [junit4]   2> 1116404 T4147 oasc.ZkController.publish publishing 
core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 1116404 T4122 oasc.ZkController.publish publishing 
core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 1116404 T4139 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 1116405 T4147 oasc.ZkController.waitForCoreNodeName look for 
our core node name
   [junit4]   2> 1116405 T4122 oasc.ZkController.waitForCoreNodeName look for 
our core node name
   [junit4]   2> 1116406 T4188 oasc.ZkController.publish publishing 
core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 1116407 T4188 oasc.ZkController.waitForCoreNodeName look for 
our core node name
   [junit4]   2> 1116408 T4131 oasc.Overseer$ClusterStateUpdater.updateState 
Update state numShards=2 message={
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "node_name":"127.0.0.1:14336_",
   [junit4]   2>          "operation":"state",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "base_url":"https://127.0.0.1:14336";,
   [junit4]   2>          "collection":"solrj_collection",
   [junit4]   2>          "core":"solrj_collection_shard1_replica1",
   [junit4]   2>          "roles":null}
   [junit4]   2> 1116411 T4131 oasc.Overseer$ClusterStateUpdater.updateState 
Update state numShards=2 message={
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "node_name":"127.0.0.1:14322_",
   [junit4]   2>          "operation":"state",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "base_url":"https://127.0.0.1:14322";,
   [junit4]   2>          "collection":"solrj_collection",
   [junit4]   2>          "core":"solrj_collection_shard1_replica2",
   [junit4]   2>          "roles":null}
   [junit4]   2> 1116413 T4139 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 1116414 T4131 oasc.Overseer$ClusterStateUpdater.updateState 
Update state numShards=2 message={
   [junit4]   2>          "shard":"shard2",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "node_name":"127.0.0.1:14317_",
   [junit4]   2>          "operation":"state",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "base_url":"https://127.0.0.1:14317";,
   [junit4]   2>          "collection":"solrj_collection",
   [junit4]   2>          "core":"solrj_collection_shard2_replica1",
   [junit4]   2>          "roles":null}
   [junit4]   2> 1116416 T4131 oasc.Overseer$ClusterStateUpdater.updateState 
Update state numShards=2 message={
   [junit4]   2>          "shard":"shard2",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "node_name":"127.0.0.1:14344_",
   [junit4]   2>          "operation":"state",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "base_url":"https://127.0.0.1:14344";,
   [junit4]   2>          "collection":"solrj_collection",
   [junit4]   2>          "core":"solrj_collection_shard2_replica2",
   [junit4]   2>          "roles":null}
   [junit4]   2> 1116418 T4139 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 1116419 T4139 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1116419 T4199 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1116419 T4154 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1116419 T4169 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1116419 T4140 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1116419 T4184 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1117404 T4173 oasc.ZkController.waitForShardId waiting to find 
shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 1117405 T4173 oasc.ZkController.createCollectionZkNode Check 
for collection zkNode:solrj_collection
   [junit4]   2> 1117405 T4173 oasc.ZkController.createCollectionZkNode 
Collection zkNode exists
   [junit4]   2> 1117405 T4147 oasc.ZkController.waitForShardId waiting to find 
shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 1117406 T4173 oascc.ZkStateReader.readConfigName Load 
collection config from:/collections/solrj_collection
   [junit4]   2> 1117406 T4147 oasc.ZkController.createCollectionZkNode Check 
for collection zkNode:solrj_collection
   [junit4]   2> 1117406 T4122 oasc.ZkController.waitForShardId waiting to find 
shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 1117407 T4147 oasc.ZkController.createCollectionZkNode 
Collection zkNode exists
   [junit4]   2> 1117407 T4173 oascc.ZkStateReader.readConfigName 
path=/collections/solrj_collection configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 1117407 T4147 oascc.ZkStateReader.readConfigName Load 
collection config from:/collections/solrj_collection
   [junit4]   2> 1117407 T4122 oasc.ZkController.createCollectionZkNode Check 
for collection zkNode:solrj_collection
   [junit4]   2> 1117407 T4188 oasc.ZkController.waitForShardId waiting to find 
shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 1117407 T4173 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-005/solrj_collection_shard1_replica1/'
   [junit4]   2> 1117408 T4188 oasc.ZkController.createCollectionZkNode Check 
for collection zkNode:solrj_collection
   [junit4]   2> 1117408 T4122 oasc.ZkController.createCollectionZkNode 
Collection zkNode exists
   [junit4]   2> 1117409 T4188 oasc.ZkController.createCollectionZkNode 
Collection zkNode exists
   [junit4]   2> 1117408 T4147 oascc.ZkStateReader.readConfigName 
path=/collections/solrj_collection configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 1117410 T4188 oascc.ZkStateReader.readConfigName Load 
collection config from:/collections/solrj_collection
   [junit4]   2> 1117409 T4122 oascc.ZkStateReader.readConfigName Load 
collection config from:/collections/solrj_collection
   [junit4]   2> 1117410 T4147 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-003/solrj_collection_shard1_replica2/'
   [junit4]   2> 1117411 T4188 oascc.ZkStateReader.readConfigName 
path=/collections/solrj_collection configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 1117411 T4122 oascc.ZkStateReader.readConfigName 
path=/collections/solrj_collection configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 1117411 T4188 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-006/solrj_collection_shard2_replica2/'
   [junit4]   2> 1117412 T4122 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-002/solrj_collection_shard2_replica1/'
   [junit4]   2> 1117472 T4173 oasc.SolrConfig.<init> Using Lucene 
MatchVersion: 5.0.0
   [junit4]   2> 1117478 T4188 oasc.SolrConfig.<init> Using Lucene 
MatchVersion: 5.0.0
   [junit4]   2> 1117484 T4122 oasc.SolrConfig.<init> Using Lucene 
MatchVersion: 5.0.0
   [junit4]   2> 1117489 T4147 oasc.SolrConfig.<init> Using Lucene 
MatchVersion: 5.0.0
   [junit4]   2> 1117520 T4188 oasc.SolrConfig.<init> Loaded SolrConfig: 
solrconfig.xml
   [junit4]   2> 1117526 T4122 oasc.SolrConfig.<init> Loaded SolrConfig: 
solrconfig.xml
   [junit4]   2> 1117527 T4122 oass.IndexSchema.readSchema Reading Solr Schema 
from schema.xml
   [junit4]   2> 1117535 T4122 oass.IndexSchema.readSchema 
[solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 1117536 T4173 oasc.SolrConfig.<init> Loaded SolrConfig: 
solrconfig.xml
   [junit4]   2> 1117557 T4147 oasc.SolrConfig.<init> Loaded SolrConfig: 
solrconfig.xml
   [junit4]   2> 1117622 T4188 oass.IndexSchema.readSchema Reading Solr Schema 
from schema.xml
   [junit4]   2> 1117633 T4188 oass.IndexSchema.readSchema 
[solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 1117637 T4173 oass.IndexSchema.readSchema Reading Solr Schema 
from schema.xml
   [junit4]   2> 1117649 T4173 oass.IndexSchema.readSchema 
[solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 1117658 T4147 oass.IndexSchema.readSchema Reading Solr Schema 
from schema.xml
   [junit4]   2> 1117666 T4147 oass.IndexSchema.readSchema 
[solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 1117886 T4122 oass.OpenExchangeRatesOrgProvider.init 
Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1117894 T4122 oass.IndexSchema.readSchema default search field 
in schema is text
   [junit4]   2> 1117896 T4122 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1117905 T4122 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 1117909 T4122 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 1117913 T4122 oass.OpenExchangeRatesOrgProvider.reload 
Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1117913 T4122 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 1117915 T4122 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 1117915 T4122 oass.OpenExchangeRatesOrgProvider.reload 
Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1117916 T4122 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 1117916 T4122 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 1117916 T4122 oasc.CoreContainer.create Creating SolrCore 
'solrj_collection_shard2_replica1' using configuration from collection 
solrj_collection
   [junit4]   2> 1117917 T4122 oasc.SolrCore.initDirectoryFactory 
org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1117917 T4122 oasc.SolrCore.<init> 
[solrj_collection_shard2_replica1] Opening new SolrCore at 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-002/solrj_collection_shard2_replica1/,
 
dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-002/solrj_collection_shard2_replica1/data/
   [junit4]   2> 1117917 T4122 oasc.JmxMonitoredMap.<init> JMX monitoring is 
enabled. Adding Solr mbeans to JMX Server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@2fd583c2
   [junit4]   2> 1117919 T4122 oasc.CachingDirectoryFactory.get return new 
directory for 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-002/solrj_collection_shard2_replica1/data
   [junit4]   2> 1117920 T4122 oasc.SolrCore.getNewIndexDir New index directory 
detected: old=null 
new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-002/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 1117921 T4122 oasc.SolrCore.initIndex WARN 
[solrj_collection_shard2_replica1] Solr index directory 
'/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-002/solrj_collection_shard2_replica1/data/index'
 doesn't exist. Creating new index...
   [junit4]   2> 1117921 T4122 oasc.CachingDirectoryFactory.get return new 
directory for 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-5E63E98220B52629-001/tempDir-002/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 1117921 T4122 oasu.RandomMergePolicy.<init> RandomMergePolicy 
wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: 
[LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=49, 
maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, 
calibrateSizeByDeletes=false, maxMergeDocs=2147483647, 
maxCFSSegmentSizeMB=2.16015625, noCFSRatio=0.8200160710875662]
   [junit4]   2> 1117922 T4122 oasc.SolrDeletionPolicy.onCommit 
SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>                
commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-6ae97533)),segFN=segments_1,generation=1}
   [junit4]   2> 1117922 T4122 oasc.SolrDeletionPolicy.updateCommits newest 
commit generation = 1
   [junit4]   2> 1117926 T4122 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1117926 T4122 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "dedupe"
   [junit4]   2> 1117926 T4122 oasup.UpdateRequestProcessorChain.init inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1117927 T4122 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1117927 T4122 oasup.UpdateRequestProcessorChain.init inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1117927 T4122 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1117927 T4122 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1117928 T4122 oasup.UpdateRequestProcessorChain.init inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 1117928 T4122 oasc.SolrCore.loadUpdateProcessorChains no 
updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1117929 T4122 oasc.RequestHandlers.initHandlersFromConfig 
created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1117929 T4122 oasc.RequestHandlers.initHandlersFromConfig 
created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1117929 T4122 oasc.RequestHandlers.initHandlersFromConfig 
created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1117930 T4122 oasc.RequestHandlers.initHandlersFromConfig 
created standard: solr.StandardRequestHandler
   [junit4]   2> 1117930 T4122 oasc.RequestHandlers.initHandlersFromConfig 
created /get: solr.RealTimeGetHandler
   [junit4]   2> 1117930 T4122 oasc.RequestHandlers.initHandlersFromConfig 
adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1117931 T4122 oasc.RequestHandlers.initHandlersFromConfig 
created /replication: solr.ReplicationHandler
   [junit4]   2> 1117931 T4122 oasc.RequestHandlers.initHandlersFromConfig 
created /update: solr.UpdateRequestHandler
   [junit4]   2> 1117931 T4122 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1117932 T4122 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1117932 T4122 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1117952 T4122 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1117954 T4122 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1117956 T4122 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1117957 T4122 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1117964 T4122 oasu.CommitTracker.<init> Hard AutoCommit: 
disabled
   [junit4]   2> 1117964 T4122 oasu.CommitTracker.<init> Soft AutoCommit: 
disabled
   [junit4]   2> 1117965 T4122 oasu.RandomMergePolicy.<init> RandomMergePolicy 
wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=39, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=0.3642578125, 
floorSegmentMB=0.9189453125, forceMergeDeletesPctAllowed=23.05856216772897, 
segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.7066579797873886
   [junit4]   2> 1117965 T4122 oasc.SolrDeletionPolicy.onInit 
SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>                
commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-6ae97533)),segFN=segments_1,generation=1}
   [junit4]   2> 1117966 T4122 oasc.SolrDeletionPolicy.updateCommits newest 
commit generation = 1
   [junit4]   2> 1117966 T4122 oass.SolrIndexSearcher.<init> Opening 
Searcher@4d9aa7f0[solrj_collection_shard2_replica1] main
   [junit4]   2> 1117966 T4122 oascc.ZkStateReader.readConfigName Load 
collection config from:/collections/solrj_collection
   [junit4]   2> 1117967 T4122 oascc.ZkStateReader.readConfigName 
path=/collections/solrj_collection configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 1117967 T4122 oasr.ManagedResourceStorage.newStorageIO Setting 
up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1117968 T4122 
oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured 
ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1117968 T4122 oasr.RestManager.init Initializing RestManager 
with initArgs: {}
   [junit4]   2> 1117968 T4122 oasr.ManagedResourceStorage.load Reading 
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1117969 T4122 
oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found 
for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1117969 T4122 oasr.ManagedResourceStorage.load Loaded null at 
path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1117969 T4122 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/_rest_managed.json
   [junit4]   2> 1117971 T4122 
oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new 
znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1117971 T4122 oasr.ManagedResourceStorage$JsonStorage.store 
Saved JSON object to path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1117971 T4122 oasr.RestManager.init Initializing 0 registered 
ManagedResources
   [junit4]   2> 1117975 T4211 oasc.SolrCore.registerSearcher 
[solrj_collection_shard2_replica1] Registered new searcher 
Searcher@4d9aa7f0[solrj_collection_shard2_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1117976 T4122 oasc.CoreContainer.registerCore registering 
core: solrj_collection_shard2_replica1
   [junit4]   2> 1117977 T4122 oasc.ZkController.register Register replica - 
core:solrj_collection_shard2_replica1 address:https://127.0.0.1:14317 
collection:solrj_collection shard:shard2
   [junit4]   2> 1117977 T4122 oascc.SolrZkClient.makePath makePath: 
/collections/solrj_collection/leader_elect/shard2/election
   [junit4]   2> 1117982 T4122 oasc.ShardLeaderElectionContext.runLeaderProcess 
Running the leader process for shard shard2
   [junit4]   2> 1117983 T4139 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 1117984 T4122 
oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see 
more replicas up for shard shard2: total=2 found=1 timeoutin=7.5050398

[...truncated too long message...]

]    >    2) Thread[id=4373, 
name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5E63E98220B52629]-EventThread,
 state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    >    3) Thread[id=5030, name=zkCallback-353-thread-4, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    4) Thread[id=4376, name=zkCallback-353-thread-1, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    5) Thread[id=4802, name=zkCallback-353-thread-3, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([5E63E98220B52629]:0)Throwable #3: 
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=5030, name=zkCallback-353-thread-4, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=4372, 
name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5E63E98220B52629]-SendThread(127.0.0.1:14310),
 state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.nio.ch.KQueueArrayWrapper.interrupt(Native 
Method)
   [junit4]    >         at 
sun.nio.ch.KQueueArrayWrapper.interrupt(KQueueArrayWrapper.java:205)
   [junit4]    >         at 
sun.nio.ch.KQueueSelectorImpl.wakeup(KQueueSelectorImpl.java:243)
   [junit4]    >         at 
java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
   [junit4]    >         at 
java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
   [junit4]    >         at 
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:102)
   [junit4]    >         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]    >    3) Thread[id=4376, name=zkCallback-353-thread-1, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    4) Thread[id=4802, name=zkCallback-353-thread-3, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([5E63E98220B52629]:0)
   [junit4] Completed on J3 in 155.67s, 1 test, 1 failure, 3 errors <<< 
FAILURES!

[...truncated 763 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:531:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:472:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:61:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/extra-targets.xml:39:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:189:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:496:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1360:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:962:
 There were test failures: 437 suites, 1795 tests, 3 suite-level errors, 1 
error, 774 ignored (7 assumptions)

Total time: 192 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #646
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 608977046 bytes
Compression is 0.0%
Took 1 min 54 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to