Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/2013/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC (asserts: false)

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicZkTest

Error Message:
SolrCore.getOpenCount()==2

Stack Trace:
java.lang.RuntimeException: SolrCore.getOpenCount()==2
        at __randomizedtesting.SeedInfo.seed([90D4927CB0B3CBDF]:0)
        at org.apache.solr.util.TestHarness.close(TestHarness.java:332)
        at org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:641)
        at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:185)
        at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicZkTest

Error Message:
Clean up static fields (in @AfterClass?), your test seems to hang on to 
approximately 14,747,808 bytes (threshold is 10,485,760). Field reference sizes 
(counted individually):   - 16,256,208 bytes, protected static 
org.apache.solr.core.SolrConfig org.apache.solr.SolrTestCaseJ4.solrConfig   - 
15,877,032 bytes, protected static 
org.apache.solr.util.TestHarness$LocalRequestFactory 
org.apache.solr.SolrTestCaseJ4.lrf   - 15,875,936 bytes, protected static 
org.apache.solr.util.TestHarness org.apache.solr.SolrTestCaseJ4.h   - 328 
bytes, private static java.util.regex.Pattern 
org.apache.solr.SolrTestCaseJ4.nonEscapedSingleQuotePattern   - 232 bytes, 
public static java.io.File org.apache.solr.cloud.AbstractZkTestCase.SOLRHOME   
- 224 bytes, private static java.util.regex.Pattern 
org.apache.solr.SolrTestCaseJ4.escapedSingleQuotePattern   - 200 bytes, 
protected static java.lang.String org.apache.solr.SolrTestCaseJ4.testSolrHome   
- 192 bytes, public static org.junit.rules.TestRule 
org.apache.solr.SolrTestCaseJ4.solrClassRules   - 128 bytes, private static 
java.lang.String org.apache.solr.SolrTestCaseJ4.factoryProp   - 72 bytes, 
protected static java.lang.String org.apache.solr.SolrTestCaseJ4.configString   
- 64 bytes, private static java.lang.String 
org.apache.solr.SolrTestCaseJ4.coreName   - 64 bytes, protected static 
java.lang.String org.apache.solr.SolrTestCaseJ4.schemaString

Stack Trace:
junit.framework.AssertionFailedError: Clean up static fields (in @AfterClass?), 
your test seems to hang on to approximately 14,747,808 bytes (threshold is 
10,485,760). Field reference sizes (counted individually):
  - 16,256,208 bytes, protected static org.apache.solr.core.SolrConfig 
org.apache.solr.SolrTestCaseJ4.solrConfig
  - 15,877,032 bytes, protected static 
org.apache.solr.util.TestHarness$LocalRequestFactory 
org.apache.solr.SolrTestCaseJ4.lrf
  - 15,875,936 bytes, protected static org.apache.solr.util.TestHarness 
org.apache.solr.SolrTestCaseJ4.h
  - 328 bytes, private static java.util.regex.Pattern 
org.apache.solr.SolrTestCaseJ4.nonEscapedSingleQuotePattern
  - 232 bytes, public static java.io.File 
org.apache.solr.cloud.AbstractZkTestCase.SOLRHOME
  - 224 bytes, private static java.util.regex.Pattern 
org.apache.solr.SolrTestCaseJ4.escapedSingleQuotePattern
  - 200 bytes, protected static java.lang.String 
org.apache.solr.SolrTestCaseJ4.testSolrHome
  - 192 bytes, public static org.junit.rules.TestRule 
org.apache.solr.SolrTestCaseJ4.solrClassRules
  - 128 bytes, private static java.lang.String 
org.apache.solr.SolrTestCaseJ4.factoryProp
  - 72 bytes, protected static java.lang.String 
org.apache.solr.SolrTestCaseJ4.configString
  - 64 bytes, private static java.lang.String 
org.apache.solr.SolrTestCaseJ4.coreName
  - 64 bytes, protected static java.lang.String 
org.apache.solr.SolrTestCaseJ4.schemaString
        at 
com.carrotsearch.randomizedtesting.rules.StaticFieldsInvariantRule$1.afterAlways(StaticFieldsInvariantRule.java:127)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicZkTest

Error Message:
17 threads leaked from SUITE scope at org.apache.solr.cloud.BasicZkTest:     1) 
Thread[id=13547, name=zkCallback-1575-thread-3, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]         at sun.misc.Unsafe.park(Native Method)         
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=13548, 
name=zkCallback-1575-thread-4, state=TIMED_WAITING, group=TGRP-BasicZkTest]     
    at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=13545, 
name=coreZkRegister-5677-thread-1, state=WAITING, group=TGRP-BasicZkTest]       
  at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=13556, 
name=Thread-6005, state=TIMED_WAITING, group=TGRP-BasicZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)   
      at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)
         at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)         
at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:793)         
at 
org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:83)
         at 
org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:310) 
        at org.apache.solr.core.Config.<init>(Config.java:121)         at 
org.apache.solr.core.Config.<init>(Config.java:91)         at 
org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:180)         at 
org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:158)     
    at 
org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:80)
         at 
org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:61)       
  at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:603)         
at org.apache.solr.handler.SolrConfigHandler$1.run(SolrConfigHandler.java:131)  
       at 
org.apache.solr.cloud.ZkController$WatcherImpl$1.run(ZkController.java:2259)    
5) Thread[id=13557, name=zkCallback-1575-thread-6, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]         at sun.misc.Unsafe.park(Native Method)         
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    6) Thread[id=13536, 
name=SUITE-BasicZkTest-seed#[90D4927CB0B3CBDF]-worker-EventThread, 
state=WAITING, group=TGRP-BasicZkTest]         at sun.misc.Unsafe.park(Native 
Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    
7) Thread[id=13538, 
name=OverseerStateUpdate-93023183709470723-192.168.57.17:_-n_0000000000, 
state=TIMED_WAITING, group=Overseer state updater.]         at 
java.lang.Object.wait(Native Method)         at 
org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:273)
         at 
org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:317)   
      at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:591) 
        at 
org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:569)         
at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:215)    
     at java.lang.Thread.run(Thread.java:745)    8) Thread[id=13543, 
name=Thread-6001, state=WAITING, group=TGRP-BasicZkTest]         at 
java.lang.Object.wait(Native Method)         at 
java.lang.Object.wait(Object.java:502)         at 
org.apache.solr.core.CloserThread.run(CoreContainer.java:929)    9) 
Thread[id=13537, name=zkCallback-1575-thread-1, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]         at sun.misc.Unsafe.park(Native Method)         
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   10) Thread[id=13549, 
name=Thread-6003, state=WAITING, group=TGRP-BasicZkTest]         at 
java.lang.Object.wait(Native Method)         at 
java.lang.Object.wait(Object.java:502)         at 
org.apache.solr.core.SolrCores.waitAddPendingCoreOps(SolrCores.java:337)        
 at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:602)         
at org.apache.solr.handler.SolrConfigHandler$1.run(SolrConfigHandler.java:131)  
       at 
org.apache.solr.cloud.ZkController$WatcherImpl$1.run(ZkController.java:2259)   
11) Thread[id=13546, name=zkCallback-1575-thread-2, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]         at sun.misc.Unsafe.park(Native Method)         
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   12) Thread[id=13555, 
name=zkCallback-1575-thread-5, state=TIMED_WAITING, group=TGRP-BasicZkTest]     
    at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   13) Thread[id=13558, 
name=Thread-6006, state=TIMED_WAITING, group=TGRP-BasicZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)   
      at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)
         at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:294)         
at 
org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:123)
         at 
org.apache.solr.core.SolrConfig.refreshRequestParams(SolrConfig.java:806)       
  at 
org.apache.solr.handler.SolrConfigHandler$1.run(SolrConfigHandler.java:118)     
    at 
org.apache.solr.cloud.ZkController$WatcherImpl$1.run(ZkController.java:2259)   
14) Thread[id=13535, 
name=SUITE-BasicZkTest-seed#[90D4927CB0B3CBDF]-worker-SendThread(127.0.0.1:53165),
 state=TIMED_WAITING, group=TGRP-BasicZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)   15) 
Thread[id=13539, 
name=OverseerCollectionProcessor-93023183709470723-192.168.57.17:_-n_0000000000,
 state=TIMED_WAITING, group=Overseer collection creation process.]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)   
      at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)
         at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)        
 at 
org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:544)
         at 
org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:302)
         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=13540, 
name=OverseerHdfsCoreFailoverThread-93023183709470723-192.168.57.17:_-n_0000000000,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=13542, 
name=searcherExecutor-5683-thread-1, state=WAITING, group=TGRP-BasicZkTest]     
    at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 17 threads leaked from 
SUITE scope at org.apache.solr.cloud.BasicZkTest: 
   1) Thread[id=13547, name=zkCallback-1575-thread-3, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=13548, name=zkCallback-1575-thread-4, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=13545, name=coreZkRegister-5677-thread-1, state=WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=13556, name=Thread-6005, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)
        at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:793)
        at 
org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:83)
        at 
org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:310)
        at org.apache.solr.core.Config.<init>(Config.java:121)
        at org.apache.solr.core.Config.<init>(Config.java:91)
        at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:180)
        at 
org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:158)
        at 
org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:80)
        at 
org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:61)
        at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:603)
        at 
org.apache.solr.handler.SolrConfigHandler$1.run(SolrConfigHandler.java:131)
        at 
org.apache.solr.cloud.ZkController$WatcherImpl$1.run(ZkController.java:2259)
   5) Thread[id=13557, name=zkCallback-1575-thread-6, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=13536, 
name=SUITE-BasicZkTest-seed#[90D4927CB0B3CBDF]-worker-EventThread, 
state=WAITING, group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   7) Thread[id=13538, 
name=OverseerStateUpdate-93023183709470723-192.168.57.17:_-n_0000000000, 
state=TIMED_WAITING, group=Overseer state updater.]
        at java.lang.Object.wait(Native Method)
        at 
org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:273)
        at 
org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:317)
        at 
org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:591)
        at 
org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:569)
        at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:215)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=13543, name=Thread-6001, state=WAITING, group=TGRP-BasicZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
   9) Thread[id=13537, name=zkCallback-1575-thread-1, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=13549, name=Thread-6003, state=WAITING, group=TGRP-BasicZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at 
org.apache.solr.core.SolrCores.waitAddPendingCoreOps(SolrCores.java:337)
        at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:602)
        at 
org.apache.solr.handler.SolrConfigHandler$1.run(SolrConfigHandler.java:131)
        at 
org.apache.solr.cloud.ZkController$WatcherImpl$1.run(ZkController.java:2259)
  11) Thread[id=13546, name=zkCallback-1575-thread-2, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=13555, name=zkCallback-1575-thread-5, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
  13) Thread[id=13558, name=Thread-6006, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)
        at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:294)
        at 
org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:123)
        at 
org.apache.solr.core.SolrConfig.refreshRequestParams(SolrConfig.java:806)
        at 
org.apache.solr.handler.SolrConfigHandler$1.run(SolrConfigHandler.java:118)
        at 
org.apache.solr.cloud.ZkController$WatcherImpl$1.run(ZkController.java:2259)
  14) Thread[id=13535, 
name=SUITE-BasicZkTest-seed#[90D4927CB0B3CBDF]-worker-SendThread(127.0.0.1:53165),
 state=TIMED_WAITING, group=TGRP-BasicZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
  15) Thread[id=13539, 
name=OverseerCollectionProcessor-93023183709470723-192.168.57.17:_-n_0000000000,
 state=TIMED_WAITING, group=Overseer collection creation process.]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)
        at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at 
org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:544)
        at 
org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:302)
        at java.lang.Thread.run(Thread.java:745)
  16) Thread[id=13540, 
name=OverseerHdfsCoreFailoverThread-93023183709470723-192.168.57.17:_-n_0000000000,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
        at java.lang.Thread.run(Thread.java:745)
  17) Thread[id=13542, name=searcherExecutor-5683-thread-1, state=WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([90D4927CB0B3CBDF]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=13547, name=zkCallback-1575-thread-3, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]         at sun.misc.Unsafe.park(Native Method)         
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=13548, 
name=zkCallback-1575-thread-4, state=TIMED_WAITING, group=TGRP-BasicZkTest]     
    at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=13545, 
name=coreZkRegister-5677-thread-1, state=WAITING, group=TGRP-BasicZkTest]       
  at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=13543, 
name=Thread-6001, state=WAITING, group=TGRP-BasicZkTest]         at 
java.lang.Object.wait(Native Method)         at 
java.lang.Object.wait(Object.java:502)         at 
org.apache.solr.core.CloserThread.run(CoreContainer.java:929)    5) 
Thread[id=13557, name=zkCallback-1575-thread-6, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]         at sun.misc.Unsafe.park(Native Method)         
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    6) Thread[id=13537, 
name=zkCallback-1575-thread-1, state=TIMED_WAITING, group=TGRP-BasicZkTest]     
    at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    7) Thread[id=13546, 
name=zkCallback-1575-thread-2, state=TIMED_WAITING, group=TGRP-BasicZkTest]     
    at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    8) Thread[id=13555, 
name=zkCallback-1575-thread-5, state=TIMED_WAITING, group=TGRP-BasicZkTest]     
    at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    9) Thread[id=13535, 
name=SUITE-BasicZkTest-seed#[90D4927CB0B3CBDF]-worker-SendThread(127.0.0.1:53165),
 state=TIMED_WAITING, group=TGRP-BasicZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)    
     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   
10) Thread[id=13540, 
name=OverseerHdfsCoreFailoverThread-93023183709470723-192.168.57.17:_-n_0000000000,
 state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=13542, 
name=searcherExecutor-5683-thread-1, state=WAITING, group=TGRP-BasicZkTest]     
    at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=13547, name=zkCallback-1575-thread-3, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=13548, name=zkCallback-1575-thread-4, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=13545, name=coreZkRegister-5677-thread-1, state=WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=13543, name=Thread-6001, state=WAITING, group=TGRP-BasicZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
   5) Thread[id=13557, name=zkCallback-1575-thread-6, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=13537, name=zkCallback-1575-thread-1, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=13546, name=zkCallback-1575-thread-2, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=13555, name=zkCallback-1575-thread-5, state=TIMED_WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=13535, 
name=SUITE-BasicZkTest-seed#[90D4927CB0B3CBDF]-worker-SendThread(127.0.0.1:53165),
 state=TIMED_WAITING, group=TGRP-BasicZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  10) Thread[id=13540, 
name=OverseerHdfsCoreFailoverThread-93023183709470723-192.168.57.17:_-n_0000000000,
 state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=13542, name=searcherExecutor-5683-thread-1, state=WAITING, 
group=TGRP-BasicZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([90D4927CB0B3CBDF]:0)




Build Log:
[...truncated 9748 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicZkTest
   [junit4]   2> Creating dataDir: 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicZkTest-90D4927CB0B3CBDF-001/init-core-data-001
   [junit4]   2> 4148507 T13520 oas.SolrTestCaseJ4.buildSSLConfig Randomized 
ssl (false) and clientAuth (false)
   [junit4]   2> 4148507 T13520 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4148509 T13521 oasc.ZkTestServer$ZKServerMain.runFromConfig 
Starting server
   [junit4]   2> 4148610 T13520 oasc.ZkTestServer.run start zk server on 
port:53165
   [junit4]   2> 4148611 T13520 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 4148612 T13520 oascc.ConnectionManager.waitForConnected 
Waiting for client to connect to ZooKeeper
   [junit4]   2> 4148619 T13528 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@2310a0a name:ZooKeeperConnection 
Watcher:127.0.0.1:53165 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 4148621 T13520 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 4148621 T13520 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 4148622 T13520 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 4148635 T13520 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 4148636 T13520 oascc.ConnectionManager.waitForConnected 
Waiting for client to connect to ZooKeeper
   [junit4]   2> 4148641 T13531 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@2511c52d 
name:ZooKeeperConnection Watcher:127.0.0.1:53165/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4148641 T13520 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 4148642 T13520 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 4148642 T13520 oascc.SolrZkClient.makePath makePath: 
/collections/collection1
   [junit4]   2> 4148651 T13520 oascc.SolrZkClient.makePath makePath: 
/collections/collection1/shards
   [junit4]   2> 4148660 T13520 oascc.SolrZkClient.makePath makePath: 
/collections/control_collection
   [junit4]   2> 4148666 T13520 oascc.SolrZkClient.makePath makePath: 
/collections/control_collection/shards
   [junit4]   2> 4148674 T13520 oasc.AbstractZkTestCase.putConfig put 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 4148674 T13520 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/solrconfig.xml
   [junit4]   2> 4148686 T13520 oasc.AbstractZkTestCase.putConfig put 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 4148687 T13520 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/schema.xml
   [junit4]   2> 4148694 T13520 oasc.AbstractZkTestCase.putConfig put 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 4148695 T13520 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 4148703 T13520 oasc.AbstractZkTestCase.putConfig put 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 4148704 T13520 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/stopwords.txt
   [junit4]   2> 4148711 T13520 oasc.AbstractZkTestCase.putConfig put 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 4148712 T13520 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/protwords.txt
   [junit4]   2> 4148720 T13520 oasc.AbstractZkTestCase.putConfig put 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 4148721 T13520 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/currency.xml
   [junit4]   2> 4148728 T13520 oasc.AbstractZkTestCase.putConfig put 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 4148728 T13520 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/enumsConfig.xml
   [junit4]   2> 4148736 T13520 oasc.AbstractZkTestCase.putConfig put 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 4148737 T13520 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/open-exchange-rates.json
   [junit4]   2> 4148743 T13520 oasc.AbstractZkTestCase.putConfig put 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 4148743 T13520 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 4148750 T13520 oasc.AbstractZkTestCase.putConfig put 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 4148751 T13520 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/old_synonyms.txt
   [junit4]   2> 4148759 T13520 oasc.AbstractZkTestCase.putConfig put 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 4148759 T13520 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/synonyms.txt
   [junit4]   2> 4148771 T13520 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 4148771 T13520 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 4148774 T13520 oasc.SolrResourceLoader.replaceClassLoader 
Adding 
'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/lib/.svn/'
 to classloader
   [junit4]   2> 4148775 T13520 oasc.SolrResourceLoader.replaceClassLoader 
Adding 
'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/lib/classes/'
 to classloader
   [junit4]   2> 4148775 T13520 oasc.SolrResourceLoader.replaceClassLoader 
Adding 
'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/lib/README'
 to classloader
   [junit4]   2> 4148872 T13520 oasc.SolrConfig.<init> Using Lucene 
MatchVersion: 6.0.0
   [junit4]   2> 4148903 T13520 oasc.SolrConfig.<init> Loaded SolrConfig: 
solrconfig.xml
   [junit4]   2> 4148904 T13520 oass.IndexSchema.readSchema Reading Solr Schema 
from 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema.xml
   [junit4]   2> 4148909 T13520 oass.IndexSchema.readSchema [null] Schema 
name=test
   [junit4]   2> 4149035 T13520 oass.OpenExchangeRatesOrgProvider.init 
Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 4149041 T13520 oass.IndexSchema.readSchema default search 
field in schema is text
   [junit4]   2> 4149043 T13520 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 4149049 T13520 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 4149051 T13520 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 4149055 T13520 oass.OpenExchangeRatesOrgProvider.reload 
Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 4149056 T13520 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 4149056 T13520 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 4149057 T13520 oass.OpenExchangeRatesOrgProvider.reload 
Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 4149057 T13520 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 4149058 T13520 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 4149058 T13520 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 4149059 T13520 oasc.SolrResourceLoader.locateSolrHome using 
system property solr.solr.home: 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr
   [junit4]   2> 4149059 T13520 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/'
   [junit4]   2> 4149092 T13520 oasc.CoreContainer.<init> New CoreContainer 
1085618884
   [junit4]   2> 4149093 T13520 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/]
   [junit4]   2> 4149093 T13520 oashc.HttpShardHandlerFactory.getParameter 
Setting socketTimeout to: 0
   [junit4]   2> 4149094 T13520 oashc.HttpShardHandlerFactory.getParameter 
Setting urlScheme to: null
   [junit4]   2> 4149094 T13520 oashc.HttpShardHandlerFactory.getParameter 
Setting connTimeout to: 0
   [junit4]   2> 4149094 T13520 oashc.HttpShardHandlerFactory.getParameter 
Setting maxConnectionsPerHost to: 20
   [junit4]   2> 4149095 T13520 oashc.HttpShardHandlerFactory.getParameter 
Setting maxConnections to: 10000
   [junit4]   2> 4149095 T13520 oashc.HttpShardHandlerFactory.getParameter 
Setting corePoolSize to: 0
   [junit4]   2> 4149095 T13520 oashc.HttpShardHandlerFactory.getParameter 
Setting maximumPoolSize to: 2147483647
   [junit4]   2> 4149095 T13520 oashc.HttpShardHandlerFactory.getParameter 
Setting maxThreadIdleTime to: 5
   [junit4]   2> 4149096 T13520 oashc.HttpShardHandlerFactory.getParameter 
Setting sizeOfQueue to: -1
   [junit4]   2> 4149096 T13520 oashc.HttpShardHandlerFactory.getParameter 
Setting fairnessPolicy to: false
   [junit4]   2> 4149096 T13520 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=30000&connTimeout=30000&retry=false
   [junit4]   2> 4149097 T13520 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 4149097 T13520 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 4149097 T13520 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 4149097 T13520 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:53165/solr
   [junit4]   2> 4149098 T13520 oasc.ZkController.checkChrootPath zkHost 
includes chroot
   [junit4]   2> 4149098 T13520 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 4149099 T13520 oascc.ConnectionManager.waitForConnected 
Waiting for client to connect to ZooKeeper
   [junit4]   2> 4149104 T13534 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@1d7fbcf name:ZooKeeperConnection 
Watcher:127.0.0.1:53165 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 4149106 T13520 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 4149106 T13520 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 4149115 T13520 oascc.ConnectionManager.waitForConnected 
Waiting for client to connect to ZooKeeper
   [junit4]   2> 4149118 T13537 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@63bbdc90 
name:ZooKeeperConnection Watcher:127.0.0.1:53165/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4149119 T13520 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 4149122 T13520 oascc.SolrZkClient.makePath makePath: 
/overseer/queue
   [junit4]   2> 4149131 T13520 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-queue-work
   [junit4]   2> 4149140 T13520 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-map-running
   [junit4]   2> 4149149 T13520 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-map-completed
   [junit4]   2> 4149159 T13520 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-map-failure
   [junit4]   2> 4149170 T13520 oascc.SolrZkClient.makePath makePath: 
/live_nodes
   [junit4]   2> 4149175 T13520 oasc.ZkController.createEphemeralLiveNode 
Register node as live in ZooKeeper:/live_nodes/192.168.57.17:_
   [junit4]   2> 4149177 T13520 oascc.SolrZkClient.makePath makePath: 
/live_nodes/192.168.57.17:_
   [junit4]   2> 4149186 T13520 oascc.SolrZkClient.makePath makePath: 
/overseer_elect
   [junit4]   2> 4149192 T13520 oascc.SolrZkClient.makePath makePath: 
/overseer_elect/election
   [junit4]   2> 4149199 T13520 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 4149203 T13520 oasc.OverseerElectionContext.runLeaderProcess I 
am going to be the leader 192.168.57.17:_
   [junit4]   2> 4149204 T13520 oascc.SolrZkClient.makePath makePath: 
/overseer_elect/leader
   [junit4]   2> 4149210 T13520 oasc.Overseer.start Overseer 
(id=93023183709470723-192.168.57.17:_-n_0000000000) starting
   [junit4]   2> 4149219 T13520 oascc.SolrZkClient.makePath makePath: 
/overseer/queue-work
   [junit4]   2> 4149242 T13520 oasc.OverseerAutoReplicaFailoverThread.<init> 
Starting OverseerAutoReplicaFailoverThread 
autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=30000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 4149243 T13539 oasc.OverseerCollectionProcessor.run Process 
current queue of collection creations
   [junit4]   2> 4149245 T13520 oascc.SolrZkClient.makePath makePath: 
/clusterstate.json
   [junit4]   2> 4149250 T13520 oascc.SolrZkClient.makePath makePath: 
/aliases.json
   [junit4]   2> 4149254 T13520 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 4149268 T13538 oasc.Overseer$ClusterStateUpdater.run Starting 
to work on the main queue
   [junit4]   2> 4149290 T13541 oasc.ZkController.publish publishing 
core=collection1 state=down collection=collection1
   [junit4]   2> 4149290 T13541 oasc.ZkController.publish numShards not found 
on descriptor - reading it from system property
   [junit4]   2> 4149292 T13537 oasc.DistributedQueue$LatchWatcher.process 
NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 4149298 T13541 oasc.ZkController.waitForShardId waiting to 
find shard id in clusterstate for collection1
   [junit4]   2> 4149299 T13538 oasc.Overseer$ClusterStateUpdater.run 
processMessage: queueSize: 0, message = {
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "core_node_name":"192.168.57.17:__collection1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"http://192.168.57.17:";,
   [junit4]   2>          "node_name":"192.168.57.17:_",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "collection":"collection1",
   [junit4]   2>          "operation":"state"} current state version: 0
   [junit4]   2> 4149300 T13538 oasco.ReplicaMutator.updateState Update state 
numShards=null message={
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "core_node_name":"192.168.57.17:__collection1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"http://192.168.57.17:";,
   [junit4]   2>          "node_name":"192.168.57.17:_",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "collection":"collection1",
   [junit4]   2>          "operation":"state"}
   [junit4]   2> 4149302 T13537 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 4150301 T13541 oasc.ZkController.createCollectionZkNode Check 
for collection zkNode:collection1
   [junit4]   2> 4150304 T13541 oasc.ZkController.createCollectionZkNode 
Collection zkNode exists
   [junit4]   2> 4150304 T13541 oascc.ZkStateReader.readConfigName Load 
collection config from:/collections/collection1
   [junit4]   2> 4150310 T13541 oascc.ZkStateReader.readConfigName 
path=/collections/collection1 configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 4150310 T13541 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 4150313 T13541 oasc.SolrResourceLoader.replaceClassLoader 
Adding 
'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/lib/.svn/'
 to classloader
   [junit4]   2> 4150313 T13541 oasc.SolrResourceLoader.replaceClassLoader 
Adding 
'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/lib/classes/'
 to classloader
   [junit4]   2> 4150314 T13541 oasc.SolrResourceLoader.replaceClassLoader 
Adding 
'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/lib/README'
 to classloader
   [junit4]   2> 4150344 T13541 oasc.ZkController.watchZKConfDir watch zkdir 
/configs/conf1
   [junit4]   2> 4150350 T13541 oasc.Config.<init> loaded config solrconfig.xml 
with version 0 
   [junit4]   2> 4150379 T13541 oasc.SolrConfig.<init> Using Lucene 
MatchVersion: 6.0.0
   [junit4]   2> 4150406 T13541 oasc.SolrConfig.<init> Loaded SolrConfig: 
solrconfig.xml
   [junit4]   2> 4150410 T13541 oass.IndexSchema.readSchema Reading Solr Schema 
from /configs/conf1/schema.xml
   [junit4]   2> 4150421 T13541 oass.IndexSchema.readSchema [collection1] 
Schema name=test
   [junit4]   2> 4150576 T13541 oass.OpenExchangeRatesOrgProvider.init 
Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 4150579 T13541 oass.IndexSchema.readSchema default search 
field in schema is text
   [junit4]   2> 4150579 T13541 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 4150612 T13541 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 4150617 T13541 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 4150621 T13541 oass.OpenExchangeRatesOrgProvider.reload 
Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 4150626 T13541 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 4150627 T13541 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 4150627 T13541 oass.OpenExchangeRatesOrgProvider.reload 
Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 4150630 T13541 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 4150630 T13541 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 4150631 T13541 oasc.CoreContainer.create Creating SolrCore 
'collection1' using configuration from collection collection1
   [junit4]   2> 4150631 T13541 oasc.SolrCore.initDirectoryFactory 
org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 4150631 T13541 oasc.SolrCore.<init> [collection1] Opening new 
SolrCore at 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/,
 
dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicZkTest-90D4927CB0B3CBDF-001/init-core-data-001/
   [junit4]   2> 4150631 T13541 oasc.JmxMonitoredMap.<init> JMX monitoring is 
enabled. Adding Solr mbeans to JMX Server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4c703985
   [junit4]   2> 4150632 T13541 oasc.CachingDirectoryFactory.get return new 
directory for 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicZkTest-90D4927CB0B3CBDF-001/init-core-data-001
   [junit4]   2> 4150632 T13541 oasc.SolrCore.getNewIndexDir New index 
directory detected: old=null 
new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicZkTest-90D4927CB0B3CBDF-001/init-core-data-001/index/
   [junit4]   2> 4150633 T13541 oasc.SolrCore.initIndex WARN [collection1] Solr 
index directory 
'/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicZkTest-90D4927CB0B3CBDF-001/init-core-data-001/index'
 doesn't exist. Creating new index...
   [junit4]   2> 4150633 T13541 oasc.CachingDirectoryFactory.get return new 
directory for 
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.BasicZkTest-90D4927CB0B3CBDF-001/init-core-data-001/index
   [junit4]   2> 4150634 T13541 oasu.RandomMergePolicy.<init> RandomMergePolicy 
wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=10, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=1.0]
   [junit4]   2> 4150634 T13541 oasc.SolrDeletionPolicy.onCommit 
SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>                
commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@3061df 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@35fc9f71)),segFN=segments_1,generation=1}
   [junit4]   2> 4150635 T13541 oasc.SolrDeletionPolicy.updateCommits newest 
commit generation = 1
   [junit4]   2> 4150635 T13541 oasc.SolrCore.initWriters created xml: 
solr.XMLResponseWriter
   [junit4]   2> 4150639 T13541 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "dedupe"
   [junit4]   2> 4150640 T13541 oasup.UpdateRequestProcessorChain.init 
inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"dedupe"
   [junit4]   2> 4150640 T13541 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 4150641 T13541 oasup.UpdateRequestProcessorChain.init 
inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"dedupe-allfields"
   [junit4]   2> 4150641 T13541 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4150641 T13541 oasup.UpdateRequestProcessorChain.init 
inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"stored_sig"
   [junit4]   2> 4150642 T13541 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 4150642 T13541 oasup.UpdateRequestProcessorChain.init 
inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"uniq-fields"
   [junit4]   2> 4150642 T13541 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 4150643 T13541 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4150643 T13541 oasup.UpdateRequestProcessorChain.init 
inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 4150644 T13541 oasc.SolrCore.loadUpdateProcessorChains no 
updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 4150644 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 4150645 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 4150646 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 4150646 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 4150647 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 4150647 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 4150648 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 4150648 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 4150649 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 4150651 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 4150651 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 4150652 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 4150652 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 4150653 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/properties: 
org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 4150653 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 4150654 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 4150654 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created standard: solr.StandardRequestHandler
   [junit4]   2> 4150654 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created dismax: solr.SearchHandler
   [junit4]   2> 4150655 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 4150656 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 4150656 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 4150657 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created defaults: solr.StandardRequestHandler
   [junit4]   2> 4150657 T13541 oasc.RequestHandlers.initHandlersFromConfig 
adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 4150657 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created lazy: solr.StandardRequestHandler
   [junit4]   2> 4150658 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4150658 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4150659 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4150659 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4150660 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created spellCheckWithWordbreak_Direct: 
org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4150660 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4150661 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4150661 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4150661 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 4150662 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 4150662 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /search-facet-def: solr.SearchHandler
   [junit4]   2> 4150663 T13541 oasc.RequestHandlers.initHandlersFromConfig 
created /search-facet-invariants: solr.SearchHandler
   [junit4]   2> 4150669 T13541 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4150671 T13541 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4150672 T13541 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4150673 T13541 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4150674 T13541 oasc.SolrCore.initDeprecatedSupport WARN 
solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config 
to use the ShowFileRequestHandler.
   [junit4]   2> 4150675 T13541 oasc.SolrCore.initStatsCache Using default 
statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 4150676 T13541 oasu.CommitTracker.<init> Hard AutoCommit: 
disabled
   [junit4]   2> 4150677 T13541 oasu.CommitTracker.<init> Soft AutoCommit: 
disabled
   [junit4]   2> 4150677 T13541 oasu.RandomMergePolicy.<init> RandomMergePolicy 
wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=41, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=1.0]
   [junit4]   2> 4150678 T13541 oasc.SolrDeletionPolicy.onInit 
SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>                
commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@3061df 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@35fc9f71)),segFN=segments_1,generation=1}
   [junit4]   2> 4150678 T13541 oasc.SolrDeletionPolicy.updateCommits newest 
commit generation = 1
   [junit4]   2> 4150678 T13541 oass.SolrIndexSearcher.<init> Opening 
Searcher@18b528f9[collection1] main
   [junit4]   2> 4150679 T13541 oascc.ZkStateReader.readConfigName Load 
collection config from:/collections/collection1
   [junit4]   2> 4150681 T13541 oascc.ZkStateReader.readConfigName 
path=/collections/collection1 configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 4150682 T13541 oasr.RestManager.init Initializing RestManager 
with initArgs: 
{storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 4150682 T13541 oasr.ManagedResourceStorage.load Reading 
_rest_managed.json using InMemoryStorage
   [junit4]   2> 4150682 T13541 oasr.ManagedResource.reloadFromStorage WARN No 
stored data found for /rest/managed
   [junit4]   2> 4150682 T13541 oasr.ManagedResourceStorage$JsonStorage.store 
Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 4150682 T13541 oasr.RestManager.init Initializing 0 registered 
ManagedResources
   [junit4]   2> 4150683 T13541 oashc.SpellCheckComponent.inform Initializing 
spell checkers
   [junit4]   2> 4150692 T13541 oass.DirectSolrSpellChecker.init init: 
{name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 4150724 T13541 oash.ReplicationHandler.inform Commits will be 
reserved for  10000
   [junit4]   2> 4150725 T13542 
oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index 
for spellchecker: default
   [junit4]   2> 4150726 T13542 
oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index 
for spellchecker: direct
   [junit4]   2> 4150726 T13542 
oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index 
for spellchecker: wordbreak
   [junit4]   2> 4150727 T13541 oasc.CoreContainer.registerCore registering 
core: collection1
   [junit4]   2> 4150727 T13542 
oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index 
for spellchecker: multipleFields
   [junit4]   2> 4150729 T13542 
oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index 
for spellchecker: jarowinkler
   [junit4]   2> 4150747 T13545 oasc.ZkController.register Register replica - 
core:collection1 address:http://192.168.57.17: collection:collection1 
shard:shard1
   [junit4]   2> 4150749 T13542 
oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index 
for spellchecker: external
   [junit4]   2> 4150750 T13542 
oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index 
for spellchecker: freq
   [junit4]   2> 4150751 T13542 
oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index 
for spellchecker: fqcn
   [junit4]   2> 4150752 T13520 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 4150753 T13542 
oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index 
for spellchecker: perDict
   [junit4]   2> 4150754 T13542 oasc.SolrCore.registerSearcher [collection1] 
Registered new searcher Searcher@18b528f9[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4150771 T13545 oascc.SolrZkClient.makePath makePath: 
/collections/collection1/leader_elect/shard1/election
   [junit4]   2> 4150782 T13545 
oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for 
shard shard1
   [junit4]   2> 4150790 T13537 oasc.DistributedQueue$LatchWatcher.process 
NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 4150792 T13545 
oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found 
to continue.
   [junit4]   2> 4150792 T13545 
oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try 
and sync
   [junit4]   2> 4150793 T13545 
oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: 
http://192.168.57.17:/collection1/ shard1
   [junit4]   2> 4150793 T13545 oascc.SolrZkClient.makePath makePath: 
/collections/collection1/leaders/shard1
   [junit4]   2> 4150799 T13520 oas.SolrTestCaseJ4.setUp ###Starting testBasic
   [junit4]   2> 4150800 T13538 oasc.Overseer$ClusterStateUpdater.run 
processMessage: queueSize: 1, message = {
   [junit4]   2>          "operation":"leader",
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "collection":"collection1"} current state version: 1
   [junit4]   2> 4150803 T13537 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 4150817 T13538 oasc.Overseer$ClusterStateUpdater.run 
processMessage: queueSize: 1, message = {
   [junit4]   2>          "operation":"leader",
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "collection":"collection1",
   [junit4]   2>          "base_url":"http://192.168.57.17:";,
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "state":"active"} current state version: 2
   [junit4]   2> 4150822 T13546 oasc.DistributedQueue$LatchWatcher.process 
NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> ASYNC  NEW_CORE C4339 name=collection1 
org.apache.solr.core.SolrCore@5039f59d url=http://192.168.57.17:/collection1 
node=192.168.57.17:_ C4339_STATE=coll:collection1 core:collection1 
props:{core=collection1, base_url=http://192.168.57.17:, 
node_name=192.168.57.17:_, state=down, leader=true}
   [junit4]   2> 4150851 T13520 C4339 P oasc.SolrCore.execute [collection1] 
webapp=null path=null 
params={q=qlkciyopsbgzyvkylsjhchghjrdf&qt=standard&start=0&rows=20&version=2.2} 
hits=0 status=0 QTime=1 
   [junit4]   2> 4150865 T13520 C4339 P oasup.LogUpdateProcessor.finish 
[collection1] webapp=null path=null params={} {delete=[42 
(-1488370941815685120)]} 0 4
   [junit4]   2> 4150867 T13545 oasc.ZkController.register We are 
http://192.168.57.17:/collection1/ and leader is 
http://192.168.57.17:/collection1/
   [junit4]   2> 4150868 T13545 oasc.ZkController.register No LogReplay needed 
for core=collection1 baseURL=http://192.168.57.17:
   [junit4]   2> 4150868 T13545 oasc.ZkController.checkRecovery WARN Skipping 
recovery according to sys prop solrcloud.skip.autorecovery
   [junit4]   2> 4150869 T13545 oasc.ZkController.publish publishing 
core=collection1 state=active collection=collection1
   [junit4]   2> 4150869 T13545 oasc.ZkController.publish numShards not found 
on descriptor - reading it from system property
   [junit4]   2> 4150870 T13520 C4339 P oasup.LogUpdateProcessor.finish 
[collection1] webapp=null path=null params={} {add=[42 (1488370941823025152)]} 
0 1
   [junit4]   2> 4150871 T13520 C4339 P oasu.DirectUpdateHandler2.commit start 
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4150874 T13537 oasc.DistributedQueue$LatchWatcher.process 
NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 4150883 T13538 oasc.Overseer$ClusterStateUpdater.run 
processMessage: queueSize: 1, message = {
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "core_node_name":"192.168.57.17:__collection1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"http://192.168.57.17:";,
   [junit4]   2>          "node_name":"192.168.57.17:_",
   [junit4]   2>          "state":"active",
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "collection":"collection1",
   [junit4]   2>          "operation":"state"} current state version: 3
   [junit4]   2> 4150887 T13538 oasco.ReplicaMutator.updateState Update state 
numShards=null message={
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "core_node_name":"192.168.57.17:__collection1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"http://192.168.57.17:";,
   [junit4]   2>          "node_name":"192.168.57.17:_",
   [junit4]   2>          "state":"active",
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "collection":"collection1",
   [junit4]   2>          "operation":"state"}
   [junit4]   2> 4150893 T13520 C4339 P oasc.SolrDeletionPolicy.onCommit 
SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2>                
commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@3061df 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@35fc9f71)),segFN=segments_1,generation=1}
   [junit4]   2>                
commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@3061df 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@35fc9f71)),segFN=segments_2,generation=2}
   [junit4]   2> 4150894 T13520 C4339 P oasc.SolrDeletionPolicy.updateCommits 
newest commit generation = 2
   [junit4]   2> 4150898 T13520 C4339 P oass.SolrIndexSearcher.<init> Opening 
Searcher@61f03941[collection1] main
   [junit4]   2> 4150899 T13520 C4339 P oasu.DirectUpdateHandler2.commit 
end_commit_flush
   [junit4]   2> 4150902 T13542 oasc.SolrCore.registerSearcher [collection1] 
Registered new searcher Searcher@61f03941[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C1)))}
   [junit4]   2> 4150903 T13520 C4339 P oasup.LogUpdateProcessor.finish 
[collection1] webapp=null path=null params={} {commit=} 0 32
   [junit4]   2> 4150922 T13520 C4339 P oasc.SolrCore.execute [collection1] 
webapp=null path=null 
params={q=id:42+AND+val_s:aa\;bb&qt=standard&start=0&rows=20&version=2.2} 
hits=1 status=0 QTime=19 
   [junit4]   2> 4150924 T13520 C4339 P oasc.SolrCore.execute [collection1] 
webapp=null path=null 
params={q=id:42+AND+val_s:"aa;bb"&qt=standard&start=0&rows=20&version=2.2} 
hits=1 status=0 QTime=0 
   [junit4]   2> 4150925 T13520 C4339 P oasc.SolrCore.execute [collection1] 
webapp=null path=null 
params={q=id:42+AND+val_s:aa&qt=standard&start=0&rows=20&version=2.2} hits=0 
status=0 QTime=1 
   [junit4]   2> 4150926 T13520 C4339 P oasup.LogUpdateProcessor.finish 
[collection1] webapp=null path=null params={} {delete=[42 
(-1488370941881745408)]} 0 1
   [junit4]   2> 4150927 T13520 C4339 P oasu.DirectUpdateHandler2.commit start 
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4150927 T13520 C4339 P oasc.SolrDeletionPolicy.onCommit 
SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2>                
commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@3061df 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@35fc9f71)),segFN=segments_2,generation=2}
   [junit4]   2>                
commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@3061df 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@35fc9f71)),segFN=segments_3,generation=3}
   [junit4]   2> 4150928 T13520 C4339 P oasc.SolrDeletionPolicy.updateCommits 
newest commit generation = 3
   [junit4]   2> 4150929 T13520 C4339 P oass.SolrIndexSearcher.<init> Opening 
Searcher@6de97b00[collection1] main
   [junit4]   2> 4150929 T13520 C4339 P oasu.DirectUpdateHandler2.commit 
end_commit_flush
   [junit4]   2> 4150930 T13542 oasc.SolrCore.registerSearcher [collection1] 
Registered new searcher Searcher@6de97b00[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4150931 T13520 C4339 P oasup.LogUpdateProcessor.finish 
[collection1] webapp=null path=null params={} {commit=} 0 4
   [junit4]   2> 4150932 T13520 C4339 P oasc.SolrCore.execute [collection1] 
webapp=null path=null params={q=id:42&qt=standard&start=0&rows=20&version=2.2} 
hits=0 status=0 QTime=0 
   [junit4]   2> 4150933 T13520 C4339 P oasup.LogUpdateProcessor.finish 
[collection1] webapp=null path=null params={} {add=[42 (1488370941889085440)]} 
0 1
   [junit4]   2> 4150934 T13520 C4339 P oasup.LogUpdateProcessor.finish 
[collection1] webapp=null path=null params={} {add=[42 (1488370941891182592)]} 
0 0
   [junit4]   2> 4150935 T13520 C4339 P oasu.DirectUpdateHandler2.commit start 
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4150939 T13520 C4339 P oasc.SolrDeletionPolicy.onCommit 
SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2>                
commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@3061df 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@35fc9f71)),segFN=segments_3,generation=3}
   [junit4]   2>                
commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@3061df 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@35fc9f71)),segFN=segments_4,generation=4}
   [junit4]   2> 4150939 T13520 C4339 P oasc.SolrDeletionPol

[...truncated too long message...]


   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=13548, name=zkCallback-1575-thread-4, 
state=TIMED_WAITING, group=TGRP-BasicZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    3) Thread[id=13545, name=coreZkRegister-5677-thread-1, 
state=WAITING, group=TGRP-BasicZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    4) Thread[id=13543, name=Thread-6001, state=WAITING, 
group=TGRP-BasicZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:502)
   [junit4]    >         at 
org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
   [junit4]    >    5) Thread[id=13557, name=zkCallback-1575-thread-6, 
state=TIMED_WAITING, group=TGRP-BasicZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    6) Thread[id=13537, name=zkCallback-1575-thread-1, 
state=TIMED_WAITING, group=TGRP-BasicZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    7) Thread[id=13546, name=zkCallback-1575-thread-2, 
state=TIMED_WAITING, group=TGRP-BasicZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    8) Thread[id=13555, name=zkCallback-1575-thread-5, 
state=TIMED_WAITING, group=TGRP-BasicZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    9) Thread[id=13535, 
name=SUITE-BasicZkTest-seed#[90D4927CB0B3CBDF]-worker-SendThread(127.0.0.1:53165),
 state=TIMED_WAITING, group=TGRP-BasicZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >   10) Thread[id=13540, 
name=OverseerHdfsCoreFailoverThread-93023183709470723-192.168.57.17:_-n_0000000000,
 state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   11) Thread[id=13542, name=searcherExecutor-5683-thread-1, 
state=WAITING, group=TGRP-BasicZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([90D4927CB0B3CBDF]:0)
   [junit4] Completed in 30.62s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 679 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:519: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:61: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:39: The 
following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:187: The 
following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:510: 
The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1349: 
The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:956: 
There were test failures: 458 suites, 1853 tests, 4 suite-level errors, 840 
ignored (18 assumptions)

Total time: 104 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0 
-XX:+UseCompressedOops -XX:+UseSerialGC (asserts: false)
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to