Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/9/

6 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
The Monkey ran for over 45 seconds and no jetties were stopped - this is worth 
investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 45 seconds and no jetties 
were stopped - this is worth investigating!
        at 
__randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5:239A3C61FD271F4D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
        at 
org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
6 threads leaked from SUITE scope at 
org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) 
Thread[id=90000, name=StoppableCommitThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)  
  2) Thread[id=90037, name=StoppableCommitThread-EventThread, state=WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    
3) Thread[id=90036, name=StoppableCommitThread-SendThread(127.0.0.1:37765), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]      
   at java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)    4) 
Thread[id=90038, name=zkCallback-3567-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    5) Thread[id=90087, 
name=zkCallback-3567-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    6) Thread[id=90086, 
name=zkCallback-3567-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE 
scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest: 
   1) Thread[id=90000, name=StoppableCommitThread, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)
   2) Thread[id=90037, name=StoppableCommitThread-EventThread, state=WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   3) Thread[id=90036, name=StoppableCommitThread-SendThread(127.0.0.1:37765), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   4) Thread[id=90038, name=zkCallback-3567-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=90087, name=zkCallback-3567-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=90086, name=zkCallback-3567-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=90036, name=StoppableCommitThread-SendThread(127.0.0.1:37765), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]      
   at java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)    
     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    
2) Thread[id=90038, name=zkCallback-3567-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    3) Thread[id=90087, 
name=zkCallback-3567-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)    4) Thread[id=90086, 
name=zkCallback-3567-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=90036, name=StoppableCommitThread-SendThread(127.0.0.1:37765), 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   2) Thread[id=90038, name=zkCallback-3567-thread-1, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=90087, name=zkCallback-3567-thread-3, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=90086, name=zkCallback-3567-thread-2, state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at __randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5]:0)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at 
__randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5:239A3C61FD271F4D]:0)
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at 
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at 
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649)
        at java.lang.StringBuilder.append(StringBuilder.java:202)
        at 
org.apache.http.client.utils.URLEncodedUtils.urlEncode(URLEncodedUtils.java:522)
        at 
org.apache.http.client.utils.URLEncodedUtils.encodeFormFields(URLEncodedUtils.java:638)
        at 
org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:390)
        at 
org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:369)
        at 
org.apache.http.client.entity.UrlEncodedFormEntity.<init>(UrlEncodedFormEntity.java:77)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:447)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:974)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:990)
        at 
org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:228)
        at 
org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:167)
        at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:668)
        at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
        at 
__randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5:239A3C61FD271F4D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:246)
        at 
org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:136)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1500018646551,generation=2,filelist=[_w3.cfe, _w3.cfs, 
_w3.si, _x5.fdt, _x5.fdx, _x5.fnm, _x5.nvd, _x5.nvm, _x5.si, _x5_Memory_0.ram, 
_x9.fdt, _x9.fdx, _x9.fnm, _x9.nvd, _x9.nvm, _x9.si, _x9_Memory_0.ram, _xb.cfe, 
_xb.cfs, _xb.si, _xc.fdt, _xc.fdx, _xc.fnm, _xc.nvd, _xc.nvm, _xc.si, 
_xc_Memory_0.ram, _xd.fdt, _xd.fdx, _xd.fnm, _xd.nvd, _xd.nvm, _xd.si, 
_xd_Memory_0.ram, _xe.fdt, _xe.fdx, _xe.fnm, _xe.nvd, _xe.nvm, _xe.si, 
_xe_Memory_0.ram, segments_2]}]> but 
was:<[{indexVersion=1500018646551,generation=2,filelist=[_w3.cfe, _w3.cfs, 
_w3.si, _x5.fdt, _x5.fdx, _x5.fnm, _x5.nvd, _x5.nvm, _x5.si, _x5_Memory_0.ram, 
_x9.fdt, _x9.fdx, _x9.fnm, _x9.nvd, _x9.nvm, _x9.si, _x9_Memory_0.ram, _xb.cfe, 
_xb.cfs, _xb.si, _xc.fdt, _xc.fdx, _xc.fnm, _xc.nvd, _xc.nvm, _xc.si, 
_xc_Memory_0.ram, _xd.fdt, _xd.fdx, _xd.fnm, _xd.nvd, _xd.nvm, _xd.si, 
_xd_Memory_0.ram, _xe.fdt, _xe.fdx, _xe.fnm, _xe.nvd, _xe.nvm, _xe.si, 
_xe_Memory_0.ram, segments_2]}, 
{indexVersion=1500018646551,generation=3,filelist=[_x4.cfe, _x4.cfs, _x4.si, 
_x5.fdt, _x5.fdx, _x5.fnm, _x5.nvd, _x5.nvm, _x5.si, _x5_Memory_0.ram, _x9.fdt, 
_x9.fdx, _x9.fnm, _x9.nvd, _x9.nvm, _x9.si, _x9_Memory_0.ram, _xb.cfe, _xb.cfs, 
_xb.si, _xc.fdt, _xc.fdx, _xc.fnm, _xc.nvd, _xc.nvm, _xc.si, _xc_Memory_0.ram, 
_xd.fdt, _xd.fdx, _xd.fnm, _xd.nvd, _xd.nvm, _xd.si, _xd_Memory_0.ram, _xe.fdt, 
_xe.fdx, _xe.fnm, _xe.nvd, _xe.nvm, _xe.si, _xe_Memory_0.ram, segments_3]}]>

Stack Trace:
java.lang.AssertionError: 
expected:<[{indexVersion=1500018646551,generation=2,filelist=[_w3.cfe, _w3.cfs, 
_w3.si, _x5.fdt, _x5.fdx, _x5.fnm, _x5.nvd, _x5.nvm, _x5.si, _x5_Memory_0.ram, 
_x9.fdt, _x9.fdx, _x9.fnm, _x9.nvd, _x9.nvm, _x9.si, _x9_Memory_0.ram, _xb.cfe, 
_xb.cfs, _xb.si, _xc.fdt, _xc.fdx, _xc.fnm, _xc.nvd, _xc.nvm, _xc.si, 
_xc_Memory_0.ram, _xd.fdt, _xd.fdx, _xd.fnm, _xd.nvd, _xd.nvm, _xd.si, 
_xd_Memory_0.ram, _xe.fdt, _xe.fdx, _xe.fnm, _xe.nvd, _xe.nvm, _xe.si, 
_xe_Memory_0.ram, segments_2]}]> but 
was:<[{indexVersion=1500018646551,generation=2,filelist=[_w3.cfe, _w3.cfs, 
_w3.si, _x5.fdt, _x5.fdx, _x5.fnm, _x5.nvd, _x5.nvm, _x5.si, _x5_Memory_0.ram, 
_x9.fdt, _x9.fdx, _x9.fnm, _x9.nvd, _x9.nvm, _x9.si, _x9_Memory_0.ram, _xb.cfe, 
_xb.cfs, _xb.si, _xc.fdt, _xc.fdx, _xc.fnm, _xc.nvd, _xc.nvm, _xc.si, 
_xc_Memory_0.ram, _xd.fdt, _xd.fdx, _xd.fnm, _xd.nvd, _xd.nvm, _xd.si, 
_xd_Memory_0.ram, _xe.fdt, _xe.fdx, _xe.fnm, _xe.nvd, _xe.nvm, _xe.si, 
_xe_Memory_0.ram, segments_2]}, 
{indexVersion=1500018646551,generation=3,filelist=[_x4.cfe, _x4.cfs, _x4.si, 
_x5.fdt, _x5.fdx, _x5.fnm, _x5.nvd, _x5.nvm, _x5.si, _x5_Memory_0.ram, _x9.fdt, 
_x9.fdx, _x9.fnm, _x9.nvd, _x9.nvm, _x9.si, _x9_Memory_0.ram, _xb.cfe, _xb.cfs, 
_xb.si, _xc.fdt, _xc.fdx, _xc.fnm, _xc.nvd, _xc.nvm, _xc.si, _xc_Memory_0.ram, 
_xd.fdt, _xd.fdx, _xd.fnm, _xd.nvd, _xd.nvm, _xd.si, _xd_Memory_0.ram, _xe.fdt, 
_xe.fdx, _xe.fnm, _xe.nvd, _xe.nvm, _xe.si, _xe_Memory_0.ram, segments_3]}]>
        at 
__randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5:8E19188B23937CB6]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at 
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1277)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12426 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/init-core-data-001
   [junit4]   2> 2896291 WARN  
(SUITE-TestReplicationHandler-seed#[ABCE03BB53DB72B5]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 2896292 INFO  
(SUITE-TestReplicationHandler-seed#[ABCE03BB53DB72B5]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2896293 INFO  
(SUITE-TestReplicationHandler-seed#[ABCE03BB53DB72B5]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 2896294 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting doTestRepeater
   [junit4]   2> 2896294 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/collection1
   [junit4]   2> 2896296 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2896297 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@2306f184{/solr,null,AVAILABLE}
   [junit4]   2> 2896297 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@1065cb39{HTTP/1.1,[http/1.1]}{127.0.0.1:41611}
   [junit4]   2> 2896298 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.Server Started @2905864ms
   [junit4]   2> 2896298 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/collection1/data,
 hostContext=/solr, hostPort=41611}
   [junit4]   2> 2896298 ERROR 
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2896298 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.1.0
   [junit4]   2> 2896298 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on 
port null
   [junit4]   2> 2896298 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config 
dir: null
   [junit4]   2> 2896298 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-07-14T07:46:53.862Z
   [junit4]   2> 2896298 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/solr.xml
   [junit4]   2> 2896302 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 2896337 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896340 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896340 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896341 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/.
   [junit4]   2> 2896341 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2896350 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 2896386 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2896396 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 2896396 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using 
configuration from instancedir 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/./collection1,
 trusted=true
   [junit4]   2> 2896396 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896396 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2896396 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/collection1],
 
dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 2896397 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.6226604577629139]
   [junit4]   2> 2896479 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2896479 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2896480 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@188a3d90
   [junit4]   2> 2896480 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@61b1c677[collection1] main]
   [junit4]   2> 2896480 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to 
use dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-001/collection1/conf
   [junit4]   2> 2896489 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following 
config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 2896489 INFO  (coreLoadExecutor-7789-thread-1) [    
x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2896490 INFO  
(searcherExecutor-7790-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@61b1c677[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2896886 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1
   [junit4]   2> 2896888 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2896889 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@79653f7e{/solr,null,AVAILABLE}
   [junit4]   2> 2896889 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@5f8fe966{HTTP/1.1,[http/1.1]}{127.0.0.1:41753}
   [junit4]   2> 2896889 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.Server Started @2906456ms
   [junit4]   2> 2896889 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1/data,
 hostContext=/solr, hostPort=41753}
   [junit4]   2> 2896890 ERROR 
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2896890 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.1.0
   [junit4]   2> 2896890 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on 
port null
   [junit4]   2> 2896890 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config 
dir: null
   [junit4]   2> 2896890 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-07-14T07:46:54.454Z
   [junit4]   2> 2896891 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/solr.xml
   [junit4]   2> 2896940 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 2896982 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896986 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896986 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2896988 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/.
   [junit4]   2> 2896988 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2896996 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 2897028 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2897029 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 2897030 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using 
configuration from instancedir 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/./collection1,
 trusted=true
   [junit4]   2> 2897030 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2897030 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2897030 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1],
 
dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 2897031 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.6226604577629139]
   [junit4]   2> 2897093 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2897093 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2897094 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@1e2a6d93
   [junit4]   2> 2897094 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@3c895466[collection1] main]
   [junit4]   2> 2897094 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to 
use dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1/conf
   [junit4]   2> 2897099 INFO  (coreLoadExecutor-7799-thread-1) [    
x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 
1000ms
   [junit4]   2> 2897100 INFO  
(searcherExecutor-7800-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@3c895466[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2897100 INFO  (qtp1630951948-121935) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 
QTime=0
   [junit4]   2> 2897100 INFO  (indexFetcher-7804-thread-1) [    x:collection1] 
o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 2897101 INFO  (indexFetcher-7804-thread-1) [    x:collection1] 
o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 2897101 INFO  (indexFetcher-7804-thread-1) [    x:collection1] 
o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 2897101 INFO  (indexFetcher-7804-thread-1) [    x:collection1] 
o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 2897554 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.AbstractConnector Stopped 
ServerConnector@5f8fe966{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2897554 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1918352296
   [junit4]   2> 2897554 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 2897555 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 2897555 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 2897555 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 2897556 INFO  (coreCloseExecutor-7805-thread-1) [    
x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@658fb64f
   [junit4]   2> 2897557 INFO  (coreCloseExecutor-7805-thread-1) [    
x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.core.collection1, tag=1703917135
   [junit4]   2> 2897558 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@79653f7e{/solr,null,UNAVAILABLE}
   [junit4]   2> 2897559 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2897559 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@78e741bd{/solr,null,AVAILABLE}
   [junit4]   2> 2897560 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@4a0127ff{HTTP/1.1,[http/1.1]}{127.0.0.1:35054}
   [junit4]   2> 2897560 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.Server Started @2907126ms
   [junit4]   2> 2897560 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1/data,
 hostContext=/solr, hostPort=35054}
   [junit4]   2> 2897560 ERROR 
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2897560 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.1.0
   [junit4]   2> 2897560 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on 
port null
   [junit4]   2> 2897560 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config 
dir: null
   [junit4]   2> 2897560 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-07-14T07:46:55.124Z
   [junit4]   2> 2897561 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/solr.xml
   [junit4]   2> 2897565 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 2897600 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2897604 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2897604 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2897605 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/.
   [junit4]   2> 2897605 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2897634 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 2897670 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2897671 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 2897671 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using 
configuration from instancedir 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/./collection1,
 trusted=true
   [junit4]   2> 2897672 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2897672 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2897684 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1],
 
dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 2897685 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.6226604577629139]
   [junit4]   2> 2897814 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2897814 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2897827 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@6deb6047
   [junit4]   2> 2897827 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@f8f0809[collection1] 
main]
   [junit4]   2> 2897827 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to 
use dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-002/collection1/conf
   [junit4]   2> 2897828 INFO  (coreLoadExecutor-7812-thread-1) [    
x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2897828 INFO  
(searcherExecutor-7813-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@f8f0809[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2898157 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/collection1
   [junit4]   2> 2898159 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2898194 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@227ab761{/solr,null,AVAILABLE}
   [junit4]   2> 2898248 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@3e8329f0{HTTP/1.1,[http/1.1]}{127.0.0.1:44104}
   [junit4]   2> 2898248 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.e.j.s.Server Started @2907815ms
   [junit4]   2> 2898248 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/collection1/data,
 hostContext=/solr, hostPort=44104}
   [junit4]   2> 2898248 ERROR 
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2898249 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.1.0
   [junit4]   2> 2898249 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on 
port null
   [junit4]   2> 2898249 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config 
dir: null
   [junit4]   2> 2898249 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-07-14T07:46:55.813Z
   [junit4]   2> 2898249 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/solr.xml
   [junit4]   2> 2898253 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 2898313 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2898317 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2898317 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2898323 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/.
   [junit4]   2> 2898323 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2898333 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
   [junit4]   2> 2898374 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2898392 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 2898392 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using 
configuration from instancedir 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/./collection1,
 trusted=true
   [junit4]   2> 2898392 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@68f7814b
   [junit4]   2> 2898392 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2898392 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/collection1],
 
dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/./collection1/data/]
   [junit4]   2> 2898393 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.6226604577629139]
   [junit4]   2> 2898435 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2898435 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2898436 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.MockRandomMergePolicy: 
org.apache.lucene.index.MockRandomMergePolicy@2d34a4f9
   [junit4]   2> 2898436 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@58c5d3c0[collection1] main]
   [junit4]   2> 2898436 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to 
use dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_ABCE03BB53DB72B5-001/solr-instance-003/collection1/conf
   [junit4]   2> 2898437 WARN  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the 
/replication suffix
   [junit4]   2> 2898439 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.h.ReplicationHandler  No value set for 'pollInterval'. 
Timer Task not started.
   [junit4]   2> 2898439 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following 
config files: schema.xml
   [junit4]   2> 2898439 INFO  (coreLoadExecutor-7822-thread-1) [    
x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2898439 INFO  
(searcherExecutor-7823-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@58c5d3c0[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2898904 INFO  (qtp1630951948-121939) [    x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update 
params={wt=javabin&version=2}{add=[0]} 0 0
   [junit4]   2> 2898904 INFO  (qtp1630951948-121932) [    x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update 
params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 2898905 INFO  (qtp1630951948-121935) [    x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update 
params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 2898905 INFO  (qtp1630951948-121936) [    x:collection1] 
o.a.s.u.DirectUpdateHandler2 start 
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2898905 INFO  (qtp1630951948-121936) [    x:collection1] 
o.a.s.u.SolrIndexWriter Calling setCommitData with 
IW:org.apache.solr.update.SolrIndexWriter@158f28cf commitCommandVersion:0
   [junit4]   2> 2898907 INFO  (qtp1630951948-121936) [    x:collection1] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@31b2070a[collection1] main]
   [junit4]   2> 2898907 INFO  (qtp1630951948-121936) [    x:collection1] 
o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2898908 INFO  
(searcherExecutor-7790-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@31b2070a[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):C3)))}
   [junit4]   2> 2898908 INFO  (qtp1630951948-121936) [    x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update 
params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=}
 0 3
   [junit4]   2> 2898910 WARN  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication 
suffix
   [junit4]   2> 2898911 INFO  (qtp1630951948-121933) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 
QTime=0
   [junit4]   2> 2898911 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Master's generation: 2
   [junit4]   2> 2898911 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Master's version: 1500018416469
   [junit4]   2> 2898911 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 2898911 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 2898911 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 2898913 INFO  (qtp1630951948-121939) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&wt=javabin&version=2&command=filelist} 
status=0 QTime=1
   [junit4]   2> 2898913 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Number of files in latest index in master: 8
   [junit4]   2> 2898914 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.0]
   [junit4]   2> 2898915 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 2898915 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Starting download (fullCopy=false) to 
MockDirectoryWrapper(RAMDirectory@742ac711 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@349b16f4)
   [junit4]   2> 2898915 INFO  (qtp1630951948-121932) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898932 INFO  (qtp1630951948-121935) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898933 INFO  (qtp1630951948-121936) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898933 INFO  (qtp1630951948-121937) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0_Memory_0.ram&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898934 INFO  (qtp1630951948-121933) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898934 INFO  (qtp1630951948-121939) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898936 INFO  (qtp1630951948-121932) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898937 INFO  (qtp1630951948-121935) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898937 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Total time taken for download 
(fullCopy=false,bytesDownloaded=1380) : 0 secs (null bytes/sec) to 
MockDirectoryWrapper(RAMDirectory@742ac711 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@349b16f4)
   [junit4]   2> 2898940 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=41, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=1.57421875, 
floorSegmentMB=1.9482421875, forceMergeDeletesPctAllowed=8.568864096386566, 
segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.39460049999639224
   [junit4]   2> 2898941 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 2898941 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@750ed32f[collection1] main]
   [junit4]   2> 2898942 INFO  
(searcherExecutor-7823-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@750ed32f[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):C3)))}
   [junit4]   2> 2898942 INFO  (qtp55242568-121991) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={wait=true&masterUrl=http://127.0.0.1:41611/solr/collection1/replication&command=fetchindex}
 status=0 QTime=33
   [junit4]   2> 2898944 INFO  (qtp55242568-121993) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/select 
params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=3 status=0 QTime=0
   [junit4]   2> 2898944 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.h.TestReplicationHandler Waited for 0ms and found 3 docs
   [junit4]   2> 2898945 WARN  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication 
suffix
   [junit4]   2> 2898946 INFO  (qtp55242568-121995) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 
QTime=0
   [junit4]   2> 2898946 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Master's generation: 2
   [junit4]   2> 2898946 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Master's version: 1500018416469
   [junit4]   2> 2898946 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 2898946 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 2898946 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 2898946 INFO  (qtp55242568-121988) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&wt=javabin&version=2&command=filelist} 
status=0 QTime=0
   [junit4]   2> 2898947 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Number of files in latest index in master: 8
   [junit4]   2> 2898948 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.0]
   [junit4]   2> 2898948 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 2898948 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Starting download (fullCopy=false) to 
MockDirectoryWrapper(RAMDirectory@7c783059 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@171da3ce)
   [junit4]   2> 2898949 INFO  (qtp55242568-121991) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898949 INFO  (qtp55242568-121989) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898950 INFO  (qtp55242568-121993) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898951 INFO  (qtp55242568-121994) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0_Memory_0.ram&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898951 INFO  (qtp55242568-121995) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898960 INFO  (qtp55242568-121988) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898961 INFO  (qtp55242568-121991) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898961 INFO  (qtp55242568-121989) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 2898962 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.h.IndexFetcher Total time taken for download 
(fullCopy=false,bytesDownloaded=1380) : 0 secs (null bytes/sec) to 
MockDirectoryWrapper(RAMDirectory@7c783059 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@171da3ce)
   [junit4]   2> 2898964 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=41, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=1.57421875, 
floorSegmentMB=1.9482421875, forceMergeDeletesPctAllowed=8.568864096386566, 
segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.39460049999639224
   [junit4]   2> 2898965 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 2898965 INFO  (explicit-fetchindex-cmd) [    x:collection1] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@68dff132[collection1] main]
   [junit4]   2> 2898966 INFO  
(searcherExecutor-7813-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@68dff132[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):C3)))}
   [junit4]   2> 2898966 INFO  (qtp343049728-121974) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={wait=true&masterUrl=http://127.0.0.1:44104/solr/collection1/replication&command=fetchindex}
 status=0 QTime=21
   [junit4]   2> 2898973 INFO  (qtp343049728-121976) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/select 
params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=3 status=0 QTime=0
   [junit4]   2> 2898973 INFO  
(TEST-TestReplicationHandler.doTestRepeater-seed#[ABCE03BB53DB72B5]) [    ] 
o.a.s.h.TestReplicationHandler Waited for 0ms and found 3 docs
   [junit4]   2> 2898973 INFO  (qtp1630951948-121938) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} 
status=0 QTime=0
   [junit4]   2> 2898974 INFO  (qtp1630951948-121937) [    x:collection1] 
o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication 
params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} 
status=0 QTime=0
   [junit4]   2> 2898975 WARN  (qtp55242568-121993) [    x:collection1] 
o.a.s.h.ReplicationHandler Exception while invoking 'details' method for 
replication on master 
   [junit4]   2> org.apache.solr.client.solrj.SolrServerException: Server 
refused connection at: http://127.0.0.1:TEST_PORT/solr
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:633)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:252)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]   2>        at 
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]   2>        at 
org.apache.solr.handler.IndexFetcher.getDetails(IndexFetcher.java:1813)
   [junit4]   2>        at 
org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:952)
   [junit4]   2>        at 
org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:330)
   [junit4]   2>        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2>        at 
org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)
   [junit4]   2>        at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
   [junit4]   2>        at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2>        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2>        at 
org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2>        at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2>        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2>        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2>        at 
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2>        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.http.conn.HttpHostConnectException: 
Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused (Connection 
refused)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
   [junit4]   2>        at 
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
   [junit4]   2>        at 
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
   [junit4]   2>        at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
   [junit4]   2>        at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
   [junit4]   2>        at 
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
   [junit4]   2>        at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   [junit4]   2>        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:523)
   [junit4]   2>        ... 36 more
   [junit4]   2> Caused by: java.net.ConnectException: Connection refused 
(Connection refused)
   [junit4]   2>        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketIm

[...truncated too long message...]

cale=und -Dtests.timezone=Asia/Dubai -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J2 | ChaosMonkeySafeLeaderWithPullReplicasTest 
(suite) <<<
   [junit4]    > Throwable #1: 
com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE 
scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest: 
   [junit4]    >    1) Thread[id=90000, name=StoppableCommitThread, 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:55)
   [junit4]    >    2) Thread[id=90037, name=StoppableCommitThread-EventThread, 
state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   [junit4]    >    3) Thread[id=90036, 
name=StoppableCommitThread-SendThread(127.0.0.1:37765), state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
   [junit4]    >    4) Thread[id=90038, name=zkCallback-3567-thread-1, 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    5) Thread[id=90087, name=zkCallback-3567-thread-3, 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    6) Thread[id=90086, name=zkCallback-3567-thread-2, 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5]:0)Throwable #2: 
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=90036, 
name=StoppableCommitThread-SendThread(127.0.0.1:37765), state=TIMED_WAITING, 
group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   [junit4]    >    2) Thread[id=90038, name=zkCallback-3567-thread-1, 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    3) Thread[id=90087, name=zkCallback-3567-thread-3, 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    4) Thread[id=90086, name=zkCallback-3567-thread-2, 
state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([ABCE03BB53DB72B5]:0)
   [junit4] Completed [502/728 (4!)] on J2 in 136.56s, 1 test, 1 failure, 2 
errors <<< FAILURES!

[...truncated 685 lines...]
   [junit4] JVM J0: stdout was not empty, see: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J0-20170714_065827_8894968092063555578472.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/heapdumps/java_pid20311.hprof
 ...
   [junit4] Heap dump file created [555493137 bytes in 8.784 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 7504 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:818:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:762:
 Some of the tests produced a heap dump, but did not fail. Maybe a suppressed 
OutOfMemoryError? Dumps created:
* java_pid20311.hprof

Total time: 467 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to