I believe I’ve seen this kind of error under disk full conditions, but I don’t 
see any other indications of that in this Jenkins log.  Maybe some other kind 
of disk problem?

FWIW (probably not much): there are 62 failures like this, but only 5 unique 
resources that aren’t accessible; each fails a different number times:

$ perl -ne 'if (/invalid api spec:/i) { s/^.*(?=invalid api spec:)//i; 
s/\s+$//; $c{$_}++; } END { for (sort {$c{$b}<=>$c{$a}} keys %c) { print 
"$c{$_}\t$_\n"; } }' consoleText
40      invalid API spec: apispec/emptySpec.json
12      invalid API spec: apispec/core.SchemaRead.json
5       invalid API spec: apispec/core.config.Commands.json
4       invalid API spec: apispec/core.SchemaRead.fields.json
1       invalid API spec: apispec/cluster.configs.delete.json

--
Steve
www.lucidworks.com

> On Jan 25, 2018, at 11:06 AM, Shalin Shekhar Mangar <shalinman...@gmail.com> 
> wrote:
> 
> This is a strange failure. The logs are full of messages "invalid API
> spec:" which is a misleading message. The message is logged if the
> call to 
> ValidatingJsonMap.class.getClassLoader().getResourceAsStream(resourceName);
> returns a null input stream. I'll fix the message but I am not sure
> why the call failed. I looked through the last two mac os failures and
> I don't see this exception. Does anyone have any ideas?
> 
> On Thu, Jan 25, 2018 at 7:53 PM, Policeman Jenkins Server
> <jenk...@thetaphi.de> wrote:
>> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4407/
>> Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC
>> 
>> 8 tests failed.
>> FAILED:  
>> org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testSpecificConfigsets
>> 
>> Error Message:
>> KeeperErrorCode = NoNode for /collections/withconfigset2
>> 
>> Stack Trace:
>> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
>> NoNode for /collections/withconfigset2
>>        at 
>> __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:530732B00F155C13]:0)
>>        at 
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
>>        at 
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
>>        at 
>> org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
>>        at 
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
>>        at 
>> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
>>        at 
>> org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testSpecificConfigsets(CollectionsAPIDistributedZkTest.java:373)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>        at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>        at java.lang.reflect.Method.invoke(Method.java:498)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>        at 
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>        at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>        at 
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>        at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>        at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>        at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>        at java.lang.Thread.run(Thread.java:748)
>> 
>> 
>> FAILED:  
>> org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionReload
>> 
>> Error Message:
>> Could not find collection : reloaded_collection
>> 
>> Stack Trace:
>> org.apache.solr.common.SolrException: Could not find collection : 
>> reloaded_collection
>>        at 
>> __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:7FC051D42DBE4980]:0)
>>        at 
>> org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
>>        at 
>> org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:247)
>>        at 
>> org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.collectStartTimes(CollectionsAPIDistributedZkTest.java:550)
>>        at 
>> org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionReload(CollectionsAPIDistributedZkTest.java:491)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>        at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>        at java.lang.reflect.Method.invoke(Method.java:498)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>        at 
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>        at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>        at 
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>        at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>        at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>        at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>        at java.lang.Thread.run(Thread.java:748)
>> 
>> 
>> FAILED:  
>> org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
>> 
>> Error Message:
>> Error from server at http://127.0.0.1:63452/solr: ADDREPLICA failed to 
>> create replica
>> 
>> Stack Trace:
>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
>> from server at http://127.0.0.1:63452/solr: ADDREPLICA failed to create 
>> replica
>>        at 
>> __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:54F9110F4FEBC6D7]:0)
>>        at 
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
>>        at 
>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
>>        at 
>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
>>        at 
>> org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
>>        at 
>> org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
>>        at 
>> org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
>>        at 
>> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
>>        at 
>> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
>>        at 
>> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
>>        at 
>> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
>>        at 
>> org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection(CollectionsAPIDistributedZkTest.java:170)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>        at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>        at java.lang.reflect.Method.invoke(Method.java:498)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>        at 
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>        at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>        at 
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>        at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>        at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>        at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>        at java.lang.Thread.run(Thread.java:748)
>> 
>> 
>> FAILED:  
>> org.apache.solr.cloud.api.collections.CustomCollectionTest.testRouteFieldForImplicitRouter
>> 
>> Error Message:
>> Collection not found: withShardField
>> 
>> Stack Trace:
>> org.apache.solr.common.SolrException: Collection not found: withShardField
>>        at 
>> __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:2B29957854C829EF]:0)
>>        at 
>> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:851)
>>        at 
>> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
>>        at 
>> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
>>        at 
>> org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
>>        at 
>> org.apache.solr.cloud.api.collections.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:142)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>        at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>        at java.lang.reflect.Method.invoke(Method.java:498)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>        at 
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>        at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>        at 
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>        at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>        at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>        at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>        at java.lang.Thread.run(Thread.java:748)
>> 
>> 
>> FAILED:  
>> org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple
>> 
>> Error Message:
>> 
>> 
>> Stack Trace:
>> java.lang.NullPointerException
>>        at 
>> __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:46CA5914DFC232CE]:0)
>>        at 
>> org.apache.solr.client.solrj.embedded.JettySolrRunner.getNodeName(JettySolrRunner.java:345)
>>        at 
>> org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple(AutoAddReplicasPlanActionTest.java:65)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>        at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>        at java.lang.reflect.Method.invoke(Method.java:498)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>        at 
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>        at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>        at 
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>        at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>        at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>        at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>        at java.lang.Thread.run(Thread.java:748)
>> 
>> 
>> FAILED:  
>> junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest
>> 
>> Error Message:
>> 10 threads leaked from SUITE scope at 
>> org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest:     1) 
>> Thread[id=25475, name=Connection evictor, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> java.lang.Thread.sleep(Native Method)         at 
>> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>>          at java.lang.Thread.run(Thread.java:748)    2) Thread[id=25489, 
>> name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), 
>> state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> java.lang.Thread.sleep(Native Method)         at 
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>          at 
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) 
>>         at 
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    3) 
>> Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> sun.misc.Unsafe.park(Native Method)         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
>>   at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>          at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>          at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)        
>>  at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)    4) Thread[id=25493, 
>> name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> sun.misc.Unsafe.park(Native Method)         at 
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>          at 
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)  
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)    5) Thread[id=25490, 
>> name=jetty-launcher-5573-thread-2-EventThread, state=WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> sun.misc.Unsafe.park(Native Method)         at 
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>          at 
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)  
>>        at 
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    6) 
>> Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> sun.misc.Unsafe.park(Native Method)         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
>>   at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>          at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>          at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)        
>>  at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)    7) Thread[id=25470, 
>> name=Connection evictor, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> java.lang.Thread.sleep(Native Method)         at 
>> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>>          at java.lang.Thread.run(Thread.java:748)    8) Thread[id=25522, 
>> name=zkCallback-5597-thread-3, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> sun.misc.Unsafe.park(Native Method)         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
>>   at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>          at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>          at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)        
>>  at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)    9) Thread[id=25524, 
>> name=zkCallback-5597-thread-5, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> sun.misc.Unsafe.park(Native Method)         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
>>   at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>          at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>          at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)        
>>  at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)   10) Thread[id=25521, 
>> name=zkCallback-5597-thread-2-processing-n:127.0.0.1:63757_solr, 
>> state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> java.lang.Thread.sleep(Native Method)         at 
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)
>>          at 
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
>>          at 
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:494)    
>>      at 
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:437)    
>>      at 
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:424)    
>>      at 
>> org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:788)
>>          at 
>> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
>>          at 
>> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135) 
>>         at 
>> org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)        
>>  at 
>> org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
>>          at 
>> org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
>>          at 
>> org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$231/1095815815.run(Unknown
>>  Source)         at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)      
>>    at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
>>          at 
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$114/259134832.run(Unknown
>>  Source)         at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)
>> 
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from 
>> SUITE scope at 
>> org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest:
>>   1) Thread[id=25475, name=Connection evictor, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]
>>        at java.lang.Thread.sleep(Native Method)
>>        at 
>> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>>        at java.lang.Thread.run(Thread.java:748)
>>   2) Thread[id=25489, 
>> name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), 
>> state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>>        at java.lang.Thread.sleep(Native Method)
>>        at 
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>        at 
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
>>        at 
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
>>   3) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]
>>        at sun.misc.Unsafe.park(Native Method)
>>        at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>        at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>>   4) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, 
>> state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>>        at sun.misc.Unsafe.park(Native Method)
>>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>        at 
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>        at 
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>>   5) Thread[id=25490, name=jetty-launcher-5573-thread-2-EventThread, 
>> state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>>        at sun.misc.Unsafe.park(Native Method)
>>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>        at 
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>        at 
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>        at 
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
>>   6) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]
>>        at sun.misc.Unsafe.park(Native Method)
>>        at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>        at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>>   7) Thread[id=25470, name=Connection evictor, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]
>>        at java.lang.Thread.sleep(Native Method)
>>        at 
>> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>>        at java.lang.Thread.run(Thread.java:748)
>>   8) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]
>>        at sun.misc.Unsafe.park(Native Method)
>>        at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>        at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>>   9) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]
>>        at sun.misc.Unsafe.park(Native Method)
>>        at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>        at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>>  10) Thread[id=25521, 
>> name=zkCallback-5597-thread-2-processing-n:127.0.0.1:63757_solr, 
>> state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>>        at java.lang.Thread.sleep(Native Method)
>>        at 
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)
>>        at 
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
>>        at 
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:494)
>>        at 
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:437)
>>        at 
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:424)
>>        at 
>> org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:788)
>>        at 
>> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
>>        at 
>> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
>>        at 
>> org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
>>        at 
>> org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
>>        at 
>> org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
>>        at 
>> org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$231/1095815815.run(Unknown
>>  Source)
>>        at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>        at 
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
>>        at 
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$114/259134832.run(Unknown
>>  Source)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>>        at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)
>> 
>> 
>> FAILED:  
>> junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest
>> 
>> Error Message:
>> There are still zombie threads that couldn't be terminated:    1) 
>> Thread[id=25489, 
>> name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), 
>> state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> java.lang.Thread.sleep(Native Method)         at 
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>          at 
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) 
>>         at 
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    2) 
>> Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> sun.misc.Unsafe.park(Native Method)         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
>>   at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>          at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>          at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)        
>>  at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)    3) Thread[id=25493, 
>> name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> sun.misc.Unsafe.park(Native Method)         at 
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>          at 
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)  
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)    4) Thread[id=25495, 
>> name=zkCallback-5597-thread-1, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> sun.misc.Unsafe.park(Native Method)         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
>>   at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>          at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>          at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)        
>>  at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)    5) Thread[id=25522, 
>> name=zkCallback-5597-thread-3, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> sun.misc.Unsafe.park(Native Method)         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
>>   at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>          at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>          at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)        
>>  at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)    6) Thread[id=25524, 
>> name=zkCallback-5597-thread-5, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> sun.misc.Unsafe.park(Native Method)         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
>>   at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>          at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>          at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)        
>>  at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)    7) Thread[id=25521, 
>> name=zkCallback-5597-thread-2, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]         at 
>> sun.misc.Unsafe.park(Native Method)         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)       
>>   at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>          at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>          at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)        
>>  at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>          at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>          at java.lang.Thread.run(Thread.java:748)
>> 
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
>> threads that couldn't be terminated:
>>   1) Thread[id=25489, 
>> name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), 
>> state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>>        at java.lang.Thread.sleep(Native Method)
>>        at 
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>        at 
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
>>        at 
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
>>   2) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]
>>        at sun.misc.Unsafe.park(Native Method)
>>        at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>        at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>>   3) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, 
>> state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>>        at sun.misc.Unsafe.park(Native Method)
>>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>        at 
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>        at 
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>>   4) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]
>>        at sun.misc.Unsafe.park(Native Method)
>>        at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>        at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>>   5) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]
>>        at sun.misc.Unsafe.park(Native Method)
>>        at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>        at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>>   6) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]
>>        at sun.misc.Unsafe.park(Native Method)
>>        at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>        at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>>   7) Thread[id=25521, name=zkCallback-5597-thread-2, state=TIMED_WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]
>>        at sun.misc.Unsafe.park(Native Method)
>>        at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>        at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>        at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>        at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>        at java.lang.Thread.run(Thread.java:748)
>>        at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)
>> 
>> 
>> FAILED:  
>> org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger
>> 
>> Error Message:
>> 
>> 
>> Stack Trace:
>> java.lang.NullPointerException
>>        at 
>> __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:C4754A65A7D93050]:0)
>>        at 
>> org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger(TriggerIntegrationTest.java:1565)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>        at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>        at java.lang.reflect.Method.invoke(Method.java:498)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>        at 
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>        at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>        at 
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>        at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>>        at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>        at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>        at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>        at 
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>        at 
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>        at 
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>        at java.lang.Thread.run(Thread.java:748)
>> 
>> 
>> 
>> 
>> Build Log:
>> [...truncated 11654 lines...]
>>   [junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
>>   [junit4]   2> Creating dataDir: 
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/init-core-data-001
>>   [junit4]   2> 16244 WARN  
>> (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [    ] 
>> o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
>>   [junit4]   2> 16244 INFO  
>> (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [    ] 
>> o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
>> w/NUMERIC_DOCVALUES_SYSPROP=false
>>   [junit4]   2> 16249 INFO  
>> (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [    ] 
>> o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
>> @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, 
>> clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
>>   [junit4]   2> 16249 INFO  
>> (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [    ] 
>> o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
>> test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
>>   [junit4]   2> 16252 INFO  
>> (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [    ] 
>> o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in 
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/tempDir-001
>>   [junit4]   2> 16267 INFO  
>> (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [    ] 
>> o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
>>   [junit4]   2> 16270 INFO  (Thread-14) [    ] o.a.s.c.ZkTestServer client 
>> port:0.0.0.0/0.0.0.0:0
>>   [junit4]   2> 16270 INFO  (Thread-14) [    ] o.a.s.c.ZkTestServer Starting 
>> server
>>   [junit4]   2> 16481 INFO  
>> (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [    ] 
>> o.a.s.c.ZkTestServer start zk server on port:53024
>>   [junit4]   2> 16566 ERROR (Thread-14) [    ] o.a.z.s.ZooKeeperServer 
>> ZKShutdownHandler is not registered, so ZooKeeper server won't take any 
>> action on ERROR or SHUTDOWN server state changes
>>   [junit4]   2> 16670 INFO  (zkConnectionManagerCallback-10-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager zkClient has connected
>>   [junit4]   2> 16754 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
>> 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
>>   [junit4]   2> 16755 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
>> 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
>>   [junit4]   2> 16757 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.e.j.s.session DefaultSessionIdManager workerName=node0
>>   [junit4]   2> 16757 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.e.j.s.session DefaultSessionIdManager workerName=node0
>>   [junit4]   2> 16757 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.e.j.s.session No SessionScavenger set, using defaults
>>   [junit4]   2> 16757 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.e.j.s.session Scavenging every 660000ms
>>   [junit4]   2> 16757 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.e.j.s.session No SessionScavenger set, using defaults
>>   [junit4]   2> 16757 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.e.j.s.session Scavenging every 660000ms
>>   [junit4]   2> 16758 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.e.j.s.h.ContextHandler Started 
>> o.e.j.s.ServletContextHandler@d63a739{/solr,null,AVAILABLE}
>>   [junit4]   2> 16758 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.e.j.s.h.ContextHandler Started 
>> o.e.j.s.ServletContextHandler@1257398a{/solr,null,AVAILABLE}
>>   [junit4]   2> 16760 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.e.j.s.AbstractConnector Started 
>> ServerConnector@356cfda{HTTP/1.1,[http/1.1]}{127.0.0.1:53029}
>>   [junit4]   2> 16761 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.e.j.s.Server Started @20200ms
>>   [junit4]   2> 16760 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.e.j.s.AbstractConnector Started 
>> ServerConnector@1e6e5b7a{HTTP/1.1,[http/1.1]}{127.0.0.1:53028}
>>   [junit4]   2> 16761 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.e.j.s.Server Started @20200ms
>>   [junit4]   2> 16761 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
>> hostPort=53029}
>>   [junit4]   2> 16761 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
>> hostPort=53028}
>>   [junit4]   2> 16761 ERROR (jetty-launcher-7-thread-1) [    ] 
>> o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
>> missing or incomplete.
>>   [junit4]   2> 16761 ERROR (jetty-launcher-7-thread-2) [    ] 
>> o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
>> missing or incomplete.
>>   [junit4]   2> 16762 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
>> 8.0.0
>>   [junit4]   2> 16762 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port 
>> null
>>   [junit4]   2> 16762 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
>>   [junit4]   2> 16762 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
>> 2018-01-25T13:19:30.715Z
>>   [junit4]   2> 16762 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
>> 8.0.0
>>   [junit4]   2> 16762 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port 
>> null
>>   [junit4]   2> 16762 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
>>   [junit4]   2> 16762 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
>> 2018-01-25T13:19:30.715Z
>>   [junit4]   2> 16762 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
>> system property or JNDI)
>>   [junit4]   2> 16766 INFO  (zkConnectionManagerCallback-13-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager zkClient has connected
>>   [junit4]   2> 16767 INFO  (zkConnectionManagerCallback-14-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager zkClient has connected
>>   [junit4]   2> 16769 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>>   [junit4]   2> 16769 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>>   [junit4]   2> 16796 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
>> o.a.z.s.NIOServerCnxn caught end of stream exception
>>   [junit4]   2> EndOfStreamException: Unable to read additional data from 
>> client sessionid 0x1612d7928310002, likely client has closed socket
>>   [junit4]   2>        at 
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
>>   [junit4]   2>        at 
>> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
>>   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
>>   [junit4]   2> 16800 INFO  (jetty-launcher-7-thread-2) [    ] 
>> o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53024/solr
>>   [junit4]   2> 16800 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
>> o.a.z.s.NIOServerCnxn caught end of stream exception
>>   [junit4]   2> EndOfStreamException: Unable to read additional data from 
>> client sessionid 0x1612d7928310001, likely client has closed socket
>>   [junit4]   2>        at 
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
>>   [junit4]   2>        at 
>> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
>>   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
>>   [junit4]   2> 16804 INFO  (jetty-launcher-7-thread-1) [    ] 
>> o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53024/solr
>>   [junit4]   2> 16842 INFO  (zkConnectionManagerCallback-20-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager zkClient has connected
>>   [junit4]   2> 16849 INFO  (zkConnectionManagerCallback-22-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager zkClient has connected
>>   [junit4]   2> 16853 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
>> o.a.z.s.NIOServerCnxn caught end of stream exception
>>   [junit4]   2> EndOfStreamException: Unable to read additional data from 
>> client sessionid 0x1612d7928310004, likely client has closed socket
>>   [junit4]   2>        at 
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
>>   [junit4]   2>        at 
>> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
>>   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
>>   [junit4]   2> 16867 INFO  
>> (zkConnectionManagerCallback-24-thread-1-processing-n:127.0.0.1:53028_solr) 
>> [n:127.0.0.1:53028_solr    ] o.a.s.c.c.ConnectionManager zkClient has 
>> connected
>>   [junit4]   2> 16868 INFO  
>> (zkConnectionManagerCallback-26-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.c.ConnectionManager zkClient has 
>> connected
>>   [junit4]   2> 17135 INFO  (jetty-launcher-7-thread-1) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>>   [junit4]   2> 17135 INFO  (jetty-launcher-7-thread-2) 
>> [n:127.0.0.1:53028_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>>   [junit4]   2> 17138 INFO  (jetty-launcher-7-thread-1) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.OverseerElectionContext I am going to 
>> be the leader 127.0.0.1:53029_solr
>>   [junit4]   2> 17139 INFO  (jetty-launcher-7-thread-2) 
>> [n:127.0.0.1:53028_solr    ] o.a.s.c.ZkController Register node as live in 
>> ZooKeeper:/live_nodes/127.0.0.1:53028_solr
>>   [junit4]   2> 17142 INFO  (jetty-launcher-7-thread-1) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.Overseer Overseer 
>> (id=99410665167454214-127.0.0.1:53029_solr-n_0000000000) starting
>>   [junit4]   2> 17173 INFO  
>> (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
>> ZooKeeper... (0) -> (1)
>>   [junit4]   2> 17173 INFO  
>> (zkCallback-23-thread-1-processing-n:127.0.0.1:53028_solr) 
>> [n:127.0.0.1:53028_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
>> ZooKeeper... (0) -> (1)
>>   [junit4]   2> 17508 INFO  (jetty-launcher-7-thread-1) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.ZkController Register node as live in 
>> ZooKeeper:/live_nodes/127.0.0.1:53029_solr
>>   [junit4]   2> 17524 INFO  
>> (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
>> ZooKeeper... (1) -> (2)
>>   [junit4]   2> 17525 INFO  
>> (zkCallback-23-thread-1-processing-n:127.0.0.1:53028_solr) 
>> [n:127.0.0.1:53028_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
>> ZooKeeper... (1) -> (2)
>>   [junit4]   2> 17573 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Adding 
>> .autoAddReplicas trigger
>>   [junit4]   2> 17579 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing 
>> /autoscaling.json with znode version 1
>>   [junit4]   2> 18329 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: 
>> [127.0.0.1:53029_solr, 127.0.0.1:53028_solr]
>>   [junit4]   2> 18329 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Current 
>> znodeVersion 1, lastZnodeVersion -1
>>   [junit4]   2> 18329 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Processed 
>> trigger updates upto znodeVersion 1
>>   [junit4]   2> 18362 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old 
>> nodeAdded markers
>>   [junit4]   2> 18365 DEBUG (ScheduledTrigger-48-thread-1) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.NodeLostTrigger Running 
>> NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
>>   [junit4]   2> 18371 INFO  (jetty-launcher-7-thread-2) 
>> [n:127.0.0.1:53028_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
>> 'solr_53028.solr.node' (registry 'solr.node') enabled at server: 
>> com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>>   [junit4]   2> 18380 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Current 
>> znodeVersion 1, lastZnodeVersion 1
>>   [junit4]   2> 18428 INFO  (jetty-launcher-7-thread-1) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
>> 'solr_53029.solr.node' (registry 'solr.node') enabled at server: 
>> com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>>   [junit4]   2> 18489 INFO  (jetty-launcher-7-thread-2) 
>> [n:127.0.0.1:53028_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
>> 'solr_53028.solr.jvm' (registry 'solr.jvm') enabled at server: 
>> com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>>   [junit4]   2> 18489 INFO  (jetty-launcher-7-thread-2) 
>> [n:127.0.0.1:53028_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
>> 'solr_53028.solr.jetty' (registry 'solr.jetty') enabled at server: 
>> com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>>   [junit4]   2> 18502 INFO  (jetty-launcher-7-thread-1) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
>> 'solr_53029.solr.jvm' (registry 'solr.jvm') enabled at server: 
>> com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>>   [junit4]   2> 18502 INFO  (jetty-launcher-7-thread-1) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
>> 'solr_53029.solr.jetty' (registry 'solr.jetty') enabled at server: 
>> com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>>   [junit4]   2> 18577 INFO  (jetty-launcher-7-thread-2) 
>> [n:127.0.0.1:53028_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
>> definitions underneath 
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/tempDir-001/node2/.
>>   [junit4]   2> 18581 INFO  (jetty-launcher-7-thread-1) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
>> definitions underneath 
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/tempDir-001/node1/.
>>   [junit4]   2> 18697 INFO  (zkConnectionManagerCallback-32-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager zkClient has connected
>>   [junit4]   2> 18704 INFO  (zkConnectionManagerCallback-36-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager zkClient has connected
>>   [junit4]   2> 18707 INFO  
>> (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [    ] 
>> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
>>   [junit4]   2> 18709 INFO  
>> (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [    ] 
>> o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53024/solr 
>> ready
>>   [junit4]   2> 18727 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTrigger
>>   [junit4]   2> 18728 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode 
>> version 2
>>   [junit4]   2> 18729 DEBUG 
>> (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing 
>> /autoscaling.json with znode version 2
>>   [junit4]   2> 18729 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Processed 
>> trigger updates upto znodeVersion 2
>>   [junit4]   2> 18734 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old 
>> nodeLost markers
>>   [junit4]   2> 18735 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old 
>> nodeAdded markers
>>   [junit4]   2> 18736 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Current 
>> znodeVersion 2, lastZnodeVersion 2
>>   [junit4]   2> 18747 INFO  (qtp657754392-72) [n:127.0.0.1:53028_solr    ] 
>> o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with 
>> params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
>>   [junit4]   2> 18777 INFO  (qtp657754392-72) [n:127.0.0.1:53028_solr    ] 
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
>> params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=30
>>   [junit4]   2> 18873 INFO  (qtp657754392-76) [n:127.0.0.1:53028_solr    ] 
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics 
>> params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core}
>>  status=0 QTime=10
>>   [junit4]   2> 18876 INFO  (qtp1778631180-70) [n:127.0.0.1:53029_solr    ] 
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics 
>> params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core}
>>  status=0 QTime=0
>>   [junit4]   2> 18877 DEBUG (qtp1778631180-66) [n:127.0.0.1:53029_solr    ] 
>> o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
>>   [junit4]   2> 18878 INFO  (qtp1778631180-66) [n:127.0.0.1:53029_solr    ] 
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling 
>> params={wt=javabin&version=2} status=0 QTime=65
>>   [junit4]   2> 18879 DEBUG 
>> (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing 
>> /autoscaling.json with znode version 3
>>   [junit4]   2> 18880 DEBUG 
>> (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: 
>> [127.0.0.1:53029_solr, 127.0.0.1:53028_solr]
>>   [junit4]   2> 18880 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Processed 
>> trigger updates upto znodeVersion 3
>>   [junit4]   2> 18883 INFO  
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.TriggerIntegrationTest 
>> TestTriggerAction init
>>   [junit4]   2> 18883 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old 
>> nodeAdded markers
>>   [junit4]   2> 18884 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Current 
>> znodeVersion 3, lastZnodeVersion 3
>>   [junit4]   2> 18884 DEBUG (ScheduledTrigger-48-thread-1) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.NodeLostTrigger Running 
>> NodeLostTrigger: node_lost_trigger with currently live nodes: 2
>>   [junit4]   2> 18884 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.e.j.s.AbstractConnector Stopped 
>> ServerConnector@1e6e5b7a{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
>>   [junit4]   2> 18885 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=950881581
>>   [junit4]   2> 18885 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.m.SolrMetricManager Closing metric reporters for 
>> registry=solr.node, tag=null
>>   [junit4]   2> 18885 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.m.r.SolrJmxReporter Closing reporter 
>> [org.apache.solr.metrics.reporters.SolrJmxReporter@1634fb6: rootName = 
>> solr_53028, domain = solr.node, service url = null, agent id = null] for 
>> registry solr.node / com.codahale.metrics.MetricRegistry@1fb41f0a
>>   [junit4]   2> 18895 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.m.SolrMetricManager Closing metric reporters for 
>> registry=solr.jvm, tag=null
>>   [junit4]   2> 18895 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.m.r.SolrJmxReporter Closing reporter 
>> [org.apache.solr.metrics.reporters.SolrJmxReporter@6b7253ba: rootName = 
>> solr_53028, domain = solr.jvm, service url = null, agent id = null] for 
>> registry solr.jvm / com.codahale.metrics.MetricRegistry@43d06aba
>>   [junit4]   2> 18900 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.m.SolrMetricManager Closing metric reporters for 
>> registry=solr.jetty, tag=null
>>   [junit4]   2> 18901 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.m.r.SolrJmxReporter Closing reporter 
>> [org.apache.solr.metrics.reporters.SolrJmxReporter@59b22ff4: rootName = 
>> solr_53028, domain = solr.jetty, service url = null, agent id = null] for 
>> registry solr.jetty / com.codahale.metrics.MetricRegistry@2116ae4f
>>   [junit4]   2> 18903 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.c.ZkController Remove node as live in 
>> ZooKeeper:/live_nodes/127.0.0.1:53028_solr
>>   [junit4]   2> 18904 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.m.SolrMetricManager Closing metric reporters for 
>> registry=solr.cluster, tag=null
>>   [junit4]   2> 18904 INFO  
>> (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
>> ZooKeeper... (2) -> (1)
>>   [junit4]   2> 18905 INFO  (zkCallback-35-thread-1) [    ] 
>> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
>>   [junit4]   2> 18904 INFO  
>> (zkCallback-23-thread-1-processing-n:127.0.0.1:53028_solr) 
>> [n:127.0.0.1:53028_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
>> ZooKeeper... (2) -> (1)
>>   [junit4]   2> 18905 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
>> 2147483647 transient cores
>>   [junit4]   2> 18907 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.e.j.s.h.ContextHandler Stopped 
>> o.e.j.s.ServletContextHandler@d63a739{/solr,null,UNAVAILABLE}
>>   [junit4]   2> 18908 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.e.j.s.session Stopped scavenging
>>   [junit4]   2> 19897 DEBUG (ScheduledTrigger-48-thread-2) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.NodeLostTrigger Running 
>> NodeLostTrigger: node_lost_trigger with currently live nodes: 1
>>   [junit4]   2> 19897 DEBUG (ScheduledTrigger-48-thread-2) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 
>> 127.0.0.1:53028_solr
>>   [junit4]   2> 20771 INFO  
>> (OverseerCollectionConfigSetProcessor-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
>> /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
>> have disconnected from ZooKeeper
>>   [junit4]   2> 20900 DEBUG (ScheduledTrigger-48-thread-2) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.NodeLostTrigger Running 
>> NodeLostTrigger: node_lost_trigger with currently live nodes: 1
>>   [junit4]   2> 21901 DEBUG (ScheduledTrigger-48-thread-2) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.NodeLostTrigger Running 
>> NodeLostTrigger: node_lost_trigger with currently live nodes: 1
>>   [junit4]   2> 22902 DEBUG (ScheduledTrigger-48-thread-2) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.NodeLostTrigger Running 
>> NodeLostTrigger: node_lost_trigger with currently live nodes: 1
>>   [junit4]   2> 22902 DEBUG (ScheduledTrigger-48-thread-2) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger 
>> firing registered processor for lost nodes: [127.0.0.1:53028_solr]
>>   [junit4]   2> 22905 DEBUG (ScheduledTrigger-48-thread-2) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown 
>> inactive - processing event: {
>>   [junit4]   2>   "id":"215c6a28112b6Tztlufkk2v8h7oh4p3og9272t",
>>   [junit4]   2>   "source":"node_lost_trigger",
>>   [junit4]   2>   "eventTime":586892827497142,
>>   [junit4]   2>   "eventType":"NODELOST",
>>   [junit4]   2>   "properties":{
>>   [junit4]   2>     "eventTimes":[586892827497142],
>>   [junit4]   2>     "nodeNames":["127.0.0.1:53028_solr"]}}
>>   [junit4]   2> 22905 DEBUG (ScheduledTrigger-48-thread-2) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.ScheduledTriggers Pausing all 
>> triggers: [node_lost_trigger]
>>   [junit4]   2> 22910 INFO  (ScheduledTrigger-48-thread-2) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.SystemLogListener Collection .system 
>> does not exist, disabling logging.
>>   [junit4]   2> 22912 DEBUG 
>> (AutoscalingActionExecutor-49-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.ScheduledTriggers -- processing 
>> actions for {
>>   [junit4]   2>   "id":"215c6a28112b6Tztlufkk2v8h7oh4p3og9272t",
>>   [junit4]   2>   "source":"node_lost_trigger",
>>   [junit4]   2>   "eventTime":586892827497142,
>>   [junit4]   2>   "eventType":"NODELOST",
>>   [junit4]   2>   "properties":{
>>   [junit4]   2>     "eventTimes":[586892827497142],
>>   [junit4]   2>     "_enqueue_time_":586895836278162,
>>   [junit4]   2>     "nodeNames":["127.0.0.1:53028_solr"]}}
>>   [junit4]   2> 22917 DEBUG 
>> (AutoscalingActionExecutor-49-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.ScheduledTriggers Resuming trigger: 
>> node_lost_trigger after 5000ms
>>   [junit4]   2> 22917 DEBUG 
>> (AutoscalingActionExecutor-49-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 
>> 6 ms for event id=215c6a28112b6Tztlufkk2v8h7oh4p3og9272t
>>   [junit4]   2> 22928 INFO  (qtp1778631180-70) [n:127.0.0.1:53029_solr    ] 
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics 
>> params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core}
>>  status=0 QTime=7
>>   [junit4]   2> 22929 DEBUG (qtp1778631180-68) [n:127.0.0.1:53029_solr    ] 
>> o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
>>   [junit4]   2> 22930 INFO  (qtp1778631180-68) [n:127.0.0.1:53029_solr    ] 
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling 
>> params={wt=javabin&version=2} status=0 QTime=12
>>   [junit4]   2> 22931 DEBUG 
>> (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing 
>> /autoscaling.json with znode version 4
>>   [junit4]   2> 22931 DEBUG 
>> (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: 
>> [127.0.0.1:53029_solr]
>>   [junit4]   2> 22932 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Processed 
>> trigger updates upto znodeVersion 4
>>   [junit4]   2> 22932 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old 
>> nodeAdded markers
>>   [junit4]   2> 22933 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Current 
>> znodeVersion 4, lastZnodeVersion 4
>>   [junit4]   2> 24933 INFO  
>> (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [  
>>   ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTrigger
>>   [junit4]   2> 24941 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.a.s.SolrTestCaseJ4 ###Starting 
>> testContinueTriggersOnOverseerRestart
>>   [junit4]   2> 24943 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode 
>> version 5
>>   [junit4]   2> 24944 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
>> 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
>>   [junit4]   2> 24945 DEBUG 
>> (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing 
>> /autoscaling.json with znode version 5
>>   [junit4]   2> 24945 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Processed 
>> trigger updates upto znodeVersion 5
>>   [junit4]   2> 24946 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
>>   [junit4]   2> 24946 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.e.j.s.session No SessionScavenger set, using defaults
>>   [junit4]   2> 24946 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.e.j.s.session Scavenging every 600000ms
>>   [junit4]   2> 24947 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.e.j.s.h.ContextHandler Started 
>> o.e.j.s.ServletContextHandler@338dbf4b{/solr,null,AVAILABLE}
>>   [junit4]   2> 24948 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.e.j.s.AbstractConnector Started 
>> ServerConnector@12a9a774{HTTP/1.1,[http/1.1]}{127.0.0.1:53065}
>>   [junit4]   2> 24948 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.e.j.s.Server Started @28387ms
>>   [junit4]   2> 24948 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
>> hostPort=53065}
>>   [junit4]   2> 24948 ERROR 
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. 
>> Logging may be missing or incomplete.
>>   [junit4]   2> 24948 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ 
>> version 8.0.0
>>   [junit4]   2> 24949 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode 
>> on port null
>>   [junit4]   2> 24949 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
>>   [junit4]   2> 24949 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
>> 2018-01-25T13:19:38.902Z
>>   [junit4]   2> 24950 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old 
>> nodeLost markers
>>   [junit4]   2> 24951 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old 
>> nodeAdded markers
>>   [junit4]   2> 24951 DEBUG 
>> (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000)
>>  [n:127.0.0.1:53029_solr    ] o.a.s.c.a.OverseerTriggerThread Current 
>> znodeVersion 5, lastZnodeVersion 5
>>   [junit4]   2> 24952 INFO  (zkConnectionManagerCallback-38-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager zkClient has connected
>>   [junit4]   2> 24953 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>>   [junit4]   2> 24964 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53024/solr
>>   [junit4]   2> 24967 INFO  (zkConnectionManagerCallback-42-thread-1) [    ] 
>> o.a.s.c.c.ConnectionManager zkClient has connected
>>   [junit4]   2> 24971 INFO  
>> (zkConnectionManagerCallback-44-thread-1-processing-n:127.0.0.1:53065_solr) 
>> [n:127.0.0.1:53065_solr    ] o.a.s.c.c.ConnectionManager zkClient has 
>> connected
>>   [junit4]   2> 24984 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [n:127.0.0.1:53065_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes 
>> from ZooKeeper... (0) -> (1)
>>   [junit4]   2> 24988 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [n:127.0.0.1:53065_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>>   [junit4]   2> 24992 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [n:127.0.0.1:53065_solr    ] o.a.s.c.ZkController Register node as live in 
>> ZooKeeper:/live_nodes/127.0.0.1:53065_solr
>>   [junit4]   2> 24993 INFO  
>> (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) 
>> [n:127.0.0.1:53029_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
>> ZooKeeper... (1) -> (2)
>>   [junit4]   2> 24993 INFO  (zkCallback-35-thread-1) [    ] 
>> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
>>   [junit4]   2> 24994 INFO  
>> (zkCallback-43-thread-1-processing-n:127.0.0.1:53065_solr) 
>> [n:127.0.0.1:53065_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
>> ZooKeeper... (1) -> (2)
>>   [junit4]   2> 25107 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [n:127.0.0.1:53065_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
>> 'solr_53065.solr.node' (registry 'solr.node') enabled at server: 
>> com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>>   [junit4]   2> 25123 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [n:127.0.0.1:53065_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
>> 'solr_53065.solr.jvm' (registry 'solr.jvm') enabled at server: 
>> com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>>   [junit4]   2> 25123 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [n:127.0.0.1:53065_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
>> 'solr_53065.solr.jetty' (registry 'solr.jetty') enabled at server: 
>> com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>>   [junit4]   2> 25125 INFO  
>> (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F])
>>  [n:127.0.0.1:53065_solr    ] o.a
>> 
>> [...truncated too long message...]
>> 
>> pache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
>>   [junit4]    >         at 
>> org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
>>   [junit4]    >         at 
>> org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$231/1095815815.run(Unknown
>>  Source)
>>   [junit4]    >         at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>   [junit4]    >         at 
>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>   [junit4]    >         at 
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
>>   [junit4]    >         at 
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$114/259134832.run(Unknown
>>  Source)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
>>   [junit4]    >        at 
>> __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)Throwable #2: 
>> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
>> threads that couldn't be terminated:
>>   [junit4]    >    1) Thread[id=25489, 
>> name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), 
>> state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>>   [junit4]    >         at java.lang.Thread.sleep(Native Method)
>>   [junit4]    >         at 
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>   [junit4]    >         at 
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
>>   [junit4]    >         at 
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
>>   [junit4]    >    2) Thread[id=25523, name=zkCallback-5597-thread-4, 
>> state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>>   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
>>   [junit4]    >         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
>>   [junit4]    >    3) Thread[id=25493, 
>> name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, 
>> group=TGRP-AutoAddReplicasPlanActionTest]
>>   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
>>   [junit4]    >         at 
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>   [junit4]    >         at 
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>   [junit4]    >         at 
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
>>   [junit4]    >    4) Thread[id=25495, name=zkCallback-5597-thread-1, 
>> state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>>   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
>>   [junit4]    >         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
>>   [junit4]    >    5) Thread[id=25522, name=zkCallback-5597-thread-3, 
>> state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>>   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
>>   [junit4]    >         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
>>   [junit4]    >    6) Thread[id=25524, name=zkCallback-5597-thread-5, 
>> state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>>   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
>>   [junit4]    >         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
>>   [junit4]    >    7) Thread[id=25521, name=zkCallback-5597-thread-2, 
>> state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>>   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
>>   [junit4]    >         at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>   [junit4]    >         at 
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>>   [junit4]    >         at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
>>   [junit4]    >        at 
>> __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)
>>   [junit4] Completed [561/768 (4!)] on J1 in 24.26s, 1 test, 3 errors <<< 
>> FAILURES!
>> 
>> [...truncated 37138 lines...]
>> -ecj-javadoc-lint-tests:
>>    [mkdir] Created dir: 
>> /var/folders/qg/h2dfw5s161s51l2bn79mrb7r0000gn/T/ecj1722876386
>> [ecj-lint] Compiling 19 source files to 
>> /var/folders/qg/h2dfw5s161s51l2bn79mrb7r0000gn/T/ecj1722876386
>> [ecj-lint] ----------
>> [ecj-lint] 1. ERROR in 
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomPlaneTest.java
>>  (at line 20)
>> [ecj-lint]     import java.util.ArrayList;
>> [ecj-lint]            ^^^^^^^^^^^^^^^^^^^
>> [ecj-lint] The import java.util.ArrayList is never used
>> [ecj-lint] ----------
>> [ecj-lint] 2. ERROR in 
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomPlaneTest.java
>>  (at line 21)
>> [ecj-lint]     import java.util.List;
>> [ecj-lint]            ^^^^^^^^^^^^^^
>> [ecj-lint] The import java.util.List is never used
>> [ecj-lint] ----------
>> [ecj-lint] 2 problems (2 errors)
>> 
>> BUILD FAILED
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:836: The 
>> following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:101: The 
>> following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build.xml:208: The 
>> following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2281:
>>  The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2112:
>>  The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2145:
>>  Compile failed; see the compiler error output for details.
>> 
>> Total time: 81 minutes 26 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> [WARNINGS] Skipping publisher since build result is FAILURE
>> Recording test results
>> Email was triggered for: Failure - Any
>> Sending email for trigger: Failure - Any
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
> 
> 
> 
> -- 
> Regards,
> Shalin Shekhar Mangar.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to