Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/741/
Java: 32bit/jdk1.8.0_172 -client -XX:+UseG1GC

3 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeLostTriggerRestoreState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at 
__randomizedtesting.SeedInfo.seed([28916BF491997653:36EBEAF0BE16383]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeLostTriggerRestoreState(TestTriggerIntegration.java:324)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at 
__randomizedtesting.SeedInfo.seed([28916BF491997653:28A7DF34B894D239]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState(TestTriggerIntegration.java:696)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeAddedTriggerRestoreState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at 
__randomizedtesting.SeedInfo.seed([28916BF491997653:A0ACE28BAB5997FE]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeAddedTriggerRestoreState(TestTriggerIntegration.java:383)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12342 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
   [junit4]   2> Creating dataDir: 
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.autoscaling.sim.TestTriggerIntegration_28916BF491997653-001\init-core-data-001
   [junit4]   2> 21912 WARN  
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=3 numCloses=3
   [junit4]   2> 21913 INFO  
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 21919 INFO  
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 21919 INFO  
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 21958 INFO  
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 22279 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 
127.0.0.1:10000_solr
   [junit4]   2> 22281 INFO  
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history 
in memory.
   [junit4] IGNOR/A 0.00s J0 | TestTriggerIntegration.testEventFromRestoredState
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | TestTriggerIntegration.testSearchRate
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 22287 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
   [junit4]   2> 22288 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 22288 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool 
executor now
   [junit4]   2> 22288 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 22289 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 22289 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread 
pool executor
   [junit4]   2> 22289 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 22289 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 22397 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 22397 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 22405 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 2
   [junit4]   2> 22429 DEBUG (simCloudManagerPool-23-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 22691 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 22691 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 22691 DEBUG (simCloudManagerPool-23-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
3
   [junit4]   2> 22698 DEBUG (simCloudManagerPool-23-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 22698 DEBUG (simCloudManagerPool-23-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
4
   [junit4]   2> 22706 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 22706 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 
127.0.0.1:10002_solr
   [junit4]   2> 22707 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 22707 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 2
   [junit4]   2> 22708 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 22709 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 22709 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 22709 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 22709 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 22711 DEBUG (ScheduledTrigger-24-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> 22712 DEBUG (ScheduledTrigger-24-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 22711 WARN  (ScheduledTrigger-24-thread-1) [    ] 
o.a.s.c.a.TriggerEventQueue Exception peeking queue of trigger 
.scheduled_maintenance
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
 ~[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
 ~[?:1.8.0_172]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.firstChild(GenericDistributedQueue.java:380)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.firstElement(GenericDistributedQueue.java:515)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:152)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.TriggerEventQueue.peekEvent(TriggerEventQueue.java:69)
 [java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:571)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_172]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 22724 WARN  (ScheduledTrigger-24-thread-1) [    ] 
o.a.s.c.a.ScheduledTrigger ScheduledTrigger ran but was already closed
   [junit4]   2> 22724 ERROR (ScheduledTrigger-24-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Unexpected exception from trigger: 
.scheduled_maintenance
   [junit4]   2> java.lang.RuntimeException: Trigger has been closed
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTrigger.run(ScheduledTrigger.java:155)
 ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:589)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_172]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 22731 DEBUG (simCloudManagerPool-23-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 22731 DEBUG (simCloudManagerPool-23-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
5
   [junit4]   2> 22732 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 22732 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 22732 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 22732 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 22732 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 22733 DEBUG (ScheduledTrigger-24-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 22733 DEBUG (ScheduledTrigger-24-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> 22754 DEBUG (ScheduledTrigger-24-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 22754 DEBUG (ScheduledTrigger-24-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> 22775 DEBUG (ScheduledTrigger-24-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 22775 DEBUG (ScheduledTrigger-24-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> 22796 DEBUG (ScheduledTrigger-24-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 22796 DEBUG (ScheduledTrigger-24-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> 22818 DEBUG (ScheduledTrigger-24-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 22818 DEBUG (ScheduledTrigger-24-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> 22837 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
   [junit4]   2> 22839 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:         2
   [junit4]   2> ## Empty nodes:        2
   [junit4]   2> ## Dead nodes:         1
   [junit4]   2> ##             127.0.0.1:10002_solr
   [junit4]   2> ## Collections:        []
   [junit4]   2> ## Max replicas per node:      0
   [junit4]   2> ## Min replicas per node:      0
   [junit4]   2> ## Total replicas:             0
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##             - autoscaling        3
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> 
   [junit4]   2> 22840 DEBUG (ScheduledTrigger-24-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 22840 DEBUG (ScheduledTrigger-24-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestTriggerIntegration 
-Dtests.method=testNodeLostTriggerRestoreState -Dtests.seed=28916BF491997653 
-Dtests.slow=true -Dtests.locale=ar -Dtests.timezone=PST -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.56s J0 | 
TestTriggerIntegration.testNodeLostTriggerRestoreState <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: The trigger did not 
fire at all
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([28916BF491997653:36EBEAF0BE16383]:0)
   [junit4]    >        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeLostTriggerRestoreState(TestTriggerIntegration.java:324)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4] IGNOR/A 0.00s J0 | TestTriggerIntegration.testEventQueue
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | 
TestTriggerIntegration.testNodeMarkersRegistration
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | TestTriggerIntegration.testListeners
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | TestTriggerIntegration.testCooldown
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | TestTriggerIntegration.testTriggerThrottling
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | TestTriggerIntegration.testNodeLostTrigger
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | TestTriggerIntegration.testNodeAddedTrigger
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4] IGNOR/A 0.00s J0 | 
TestTriggerIntegration.testNodeAddedTriggerRestoreState
   [junit4]    > Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 22844 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 22844 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 22845 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 22845 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 22845 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool 
executor
   [junit4]   2> 22845 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 22845 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> NOTE: leaving temporary files on disk at: 
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.autoscaling.sim.TestTriggerIntegration_28916BF491997653-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
docValues:{}, maxPointsInLeafNode=1505, maxMBSortInHeap=6.6407289666570914, 
sim=RandomSimilarity(queryNorm=false): {}, locale=ar, timezone=PST
   [junit4]   2> NOTE: Windows 10 10.0 x86/Oracle Corporation 1.8.0_172 
(32-bit)/cpus=3,threads=1,free=26866208,total=61865984
   [junit4]   2> NOTE: All tests run in this JVM: 
[TlogReplayBufferedWhileIndexingTest, IndexSchemaTest, 
HdfsUnloadDistributedZkTest, TestTriggerIntegration]
   [junit4] Completed [4/828 (1!)] on J0 in 0.97s, 11 tests, 1 failure, 10 
skipped <<< FAILURES!

[...truncated 47805 lines...]
[repro] Jenkins log URL: 
https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/741/consoleText

[repro] Revision: 5a7cb240652756bcd4d28231797e053e8bb40ec7

[repro] Ant options: "-Dargs=-client -XX:+UseG1GC"

[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr\core
[repro]       TestTriggerIntegration
[repro] ant compile-test

[...truncated 3430 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.TestTriggerIntegration" -Dtests.showOutput=onerror 
"-Dargs=-client -XX:+UseG1GC" -Dtests.seed=28916BF491997653 -Dtests.slow=true 
-Dtests.locale=ar -Dtests.timezone=PST -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1

[...truncated 32 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
   [junit4]   2> Creating dataDir: 
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.autoscaling.sim.TestTriggerIntegration_28916BF491997653-001\init-core-data-001
   [junit4]   2> 15097 INFO  
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 15393 INFO  
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.e.j.u.log Logging initialized @15447ms to org.eclipse.jetty.util.log.Slf4jLog
   [junit4]   2> 15431 INFO  
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 15478 INFO  
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 15867 INFO  
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 16779 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 
127.0.0.1:10000_solr
   [junit4]   2> 16976 INFO  
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history 
in memory.
   [junit4]   2> 17039 INFO  
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[28916BF491997653])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventFromRestoredState
   [junit4]   2> 17042 INFO  
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 17052 DEBUG 
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool 
executor now
   [junit4]   2> 17053 DEBUG 
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 17053 DEBUG 
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 17053 DEBUG 
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread 
pool executor
   [junit4]   2> 17053 DEBUG 
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 17053 DEBUG 
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 17097 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 17100 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 17115 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 2
   [junit4]   2> 17854 DEBUG (simCloudManagerPool-7-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 17924 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
   [junit4]   2> 17924 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 17930 DEBUG (simCloudManagerPool-7-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
3
   [junit4]   2> 18065 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 18066 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 18067 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 2
   [junit4]   2> 18067 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 18070 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 18070 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 18077 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18099 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18075 WARN  (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.TriggerEventQueue Exception peeking queue of trigger 
.scheduled_maintenance
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
 ~[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
 ~[?:1.8.0_172]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.firstChild(GenericDistributedQueue.java:380)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.firstElement(GenericDistributedQueue.java:515)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:152)
 ~[test/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.TriggerEventQueue.peekEvent(TriggerEventQueue.java:69)
 [java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:571)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_172]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 18105 WARN  (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.ScheduledTrigger ScheduledTrigger ran but was already closed
   [junit4]   2> 18105 ERROR (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers Unexpected exception from trigger: 
.scheduled_maintenance
   [junit4]   2> java.lang.RuntimeException: Trigger has been closed
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTrigger.run(ScheduledTrigger.java:155)
 ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:589)
 [java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_172]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 18113 DEBUG (simCloudManagerPool-7-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 18114 DEBUG (simCloudManagerPool-7-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
4
   [junit4]   2> 18115 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 18116 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 18116 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 18116 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 18117 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 18117 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 18118 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 18122 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 18140 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 18141 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 18141 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10002_solr at time 
532293736065749
   [junit4]   2> 18143 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 18163 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 18163 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 18165 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 18185 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 18185 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 18187 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 18205 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 18205 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 18209 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 18227 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 18227 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 18229 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 18249 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 18249 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 18250 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 18270 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 18270 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 18271 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 18300 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 18300 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 18307 DEBUG (ScheduledTrigger-8-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 18321 DEBUG (ScheduledTrigger-8-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 18329 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 18329 DEBUG (ScheduledTrigger-8-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 18342 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 18352 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 18352 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 18352 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10002_solr] added at times 
[532293736065749], now=532304277900699
   [junit4]   2> 18353 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1e41e4b0dead5Tckrf5gmjxyt4s1w0wtuzb514v",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":532293736065749,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[532293736065749],
   [junit4]   2>     "nodeNames":["127.0.0.1:10002_solr"]}}
   [junit4]   2> 18354 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_trigger]
   [junit4]   2> 18523 INFO  
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[28916BF491997653])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testEventFromRestoredState
   [junit4]   2> 18599 INFO  
(TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:         3
   [junit4]   2> ## Empty nodes:        3
   [junit4]   2> ## Dead nodes:         0
   [junit4]   2> ## Collections:        []
   [junit4]   2> ## Max replicas per node:      0
   [junit4]   2> ## Min replicas per node:      0
   [junit4]   2> ## Total replicas:             0
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##             - autoscaling        2
   [junit4]   2> ##             - update             1
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestTriggerIntegration -Dtests.method=testEventFromRestoredState 
-Dtests.seed=28916BF491997653 -Dtests.slow=true -Dtests.badapples=true 
-Dtests.locale=ar -Dtests.timezone=PST -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 1.64s J0 | 
TestTriggerIntegration.testEventFromRestoredState <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: The trigger did not 
fire at all
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([28916BF491997653:28A7DF34B894D239]:0)
   [junit4]    >        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState(TestTriggerIntegration.java:696)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 18649 DEBUG (simCloudManagerPool-7-thread-3) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- no leader in shard1:{
   [junit4]   2>   "replicas":{"core_node1":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10000_solr",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 18657 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testSearchRate
   [junit4]   2> 18658 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
5
   [junit4]   2> 18658 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 
127.0.0.1:10001_solr
   [junit4]   2> 18659 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.c.a.s.SimNodeStateProvider --removing value for 127.0.0.1:10000_solr
   [junit4]   2> 18659 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 18659 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 18659 DEBUG (ScheduledTrigger-8-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - removing 
marker path: /autoscaling/nodeAdded/127.0.0.1:10002_solr
   [junit4]   2> 18660 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 18660 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 18660 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 18661 DEBUG (AutoscalingActionExecutor-9-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1e41e4b0dead5Tckrf5gmjxyt4s1w0wtuzb514v",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":532293736065749,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[532293736065749],
   [junit4]   2>     "_enqueue_time_":532304437435749,
   [junit4]   2>     "nodeNames":["127.0.0.1:10002_solr"]}}
   [junit4]   2> 18665 WARN  (AutoscalingActionExecutor-9-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: 
ScheduledTrigger node_added_trigger has been closed.
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.dequeue(ScheduledTriggers.java:546)
 ~[java/:?]
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:319)
 ~[java/:?]
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
 ~[java/:?]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_172]
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_172]
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
   [junit4]   2> 18670 DEBUG (AutoscalingActionExecutor-9-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 432 ms for event 
id=1e41e4b0dead5Tckrf5gmjxyt4s1w0wtuzb514v
   [junit4]   2> 18670 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool 
executor
   [junit4]   2> 18670 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 18670 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 18672 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 18673 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 6
   [junit4]   2> 18673 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion -1
   [junit4]   2> 18674 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 18674 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 18674 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 18675 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 18675 DEBUG (ScheduledTrigger-11-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18696 DEBUG (ScheduledTrigger-11-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18720 DEBUG (ScheduledTrigger-11-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18741 DEBUG (ScheduledTrigger-11-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18762 DEBUG (ScheduledTrigger-11-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18783 DEBUG (ScheduledTrigger-11-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18804 DEBUG (ScheduledTrigger-11-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18818 DEBUG (simCloudManagerPool-10-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 18819 DEBUG (simCloudManagerPool-10-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
7
   [junit4]   2> 18822 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 18822 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 18822 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 18824 DEBUG (ScheduledTrigger-11-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18846 DEBUG (ScheduledTrigger-11-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18867 DEBUG (ScheduledTrigger-11-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18887 DEBUG (ScheduledTrigger-11-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18898 DEBUG (simCloudManagerPool-10-thread-6) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 18900 DEBUG (simCloudManagerPool-10-thread-6) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
8
   [junit4]   2> 18901 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 18902 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 18902 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 18906 DEBUG (simCloudManagerPool-10-thread-7) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 18906 DEBUG (simCloudManagerPool-10-thread-7) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
9
   [junit4]   2> 18908 DEBUG (ScheduledTrigger-11-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18909 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 18911 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 18911 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 18930 DEBUG (ScheduledTrigger-11-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18951 DEBUG (ScheduledTrigger-11-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18973 DEBUG (ScheduledTrigger-11-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 18973 DEBUG (ScheduledTrigger-11-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1e4274deadcc3Tckrf5gmjxyt4s1w0wtuzb514x",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":532332438805699,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       "{\"core_node2\":{\n    
\"core\":\"collection1_shard1_replica_n2\",\n    
\"SEARCHER.searcher.maxDoc\":0,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    
\"INDEX.sizeInBytes\":10240,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250,\n    
\"INDEX.sizeInGB\":9.5367431640625E-6,\n    \"SEARCHER.searcher.numDocs\":0,\n  
  \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node1\":{\n    
\"core\":\"collection1_shard1_replica_n1\",\n    
\"SEARCHER.searcher.maxDoc\":0,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    
\"INDEX.sizeInBytes\":10240,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10002_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250,\n    
\"INDEX.sizeInGB\":9.5367431640625E-6,\n    \"SEARCHER.searcher.numDocs\":0,\n  
  \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"collection1":{"shard1":500.0}},
   [junit4]   2>     "hotCollections":{"collection1":500.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10002_solr":250.0},
   [junit4]   2>     "requestedOps":[
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  
\"first\":\"collection1\",\n  \"second\":\"shard1\"}]}}",
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  
\"first\":\"collection1\",\n  \"second\":\"shard1\"}]}}",
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  
\"first\":\"collection1\",\n  \"second\":\"shard1\"}]}}"]}}
   [junit4]   2> 18976 DEBUG (ScheduledTrigger-11-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
search_rate_trigger]
   [junit4]   2> 19015 DEBUG (simCloudManagerPool-10-thread-8) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- no leader in shard1:{
   [junit4]   2>   "replicas":{"core_node2":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10001_solr",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 19017 DEBUG (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1e4274deadcc3Tckrf5gmjxyt4s1w0wtuzb514x",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":532332438805699,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       "{\"core_node2\":{\n    
\"core\":\"collection1_shard1_replica_n2\",\n    
\"SEARCHER.searcher.maxDoc\":0,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    
\"INDEX.sizeInBytes\":10240,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250,\n    
\"INDEX.sizeInGB\":9.5367431640625E-6,\n    \"SEARCHER.searcher.numDocs\":0,\n  
  \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node1\":{\n    
\"core\":\"collection1_shard1_replica_n1\",\n    
\"SEARCHER.searcher.maxDoc\":0,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    
\"INDEX.sizeInBytes\":10240,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10002_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250,\n    
\"INDEX.sizeInGB\":9.5367431640625E-6,\n    \"SEARCHER.searcher.numDocs\":0,\n  
  \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"collection1":{"shard1":500.0}},
   [junit4]   2>     "hotCollections":{"collection1":500.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10002_solr":250.0},
   [junit4]   2>     "_enqueue_time_":532335566770699,
   [junit4]   2>     "requestedOps":[
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  
\"first\":\"collection1\",\n  \"second\":\"shard1\"}]}}",
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  
\"first\":\"collection1\",\n  \"second\":\"shard1\"}]}}",
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  
\"first\":\"collection1\",\n  \"second\":\"shard1\"}]}}"]}}
   [junit4]   2> 19017 DEBUG (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.a.ComputePlanAction -- processing event: {
   [junit4]   2>   "id":"1e4274deadcc3Tckrf5gmjxyt4s1w0wtuzb514x",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":532332438805699,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       "{\"core_node2\":{\n    
\"core\":\"collection1_shard1_replica_n2\",\n    
\"SEARCHER.searcher.maxDoc\":0,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    
\"INDEX.sizeInBytes\":10240,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250,\n    
\"INDEX.sizeInGB\":9.5367431640625E-6,\n    \"SEARCHER.searcher.numDocs\":0,\n  
  \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node1\":{\n    
\"core\":\"collection1_shard1_replica_n1\",\n    
\"SEARCHER.searcher.maxDoc\":0,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    
\"INDEX.sizeInBytes\":10240,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10002_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250,\n    
\"INDEX.sizeInGB\":9.5367431640625E-6,\n    \"SEARCHER.searcher.numDocs\":0,\n  
  \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"collection1":{"shard1":500.0}},
   [junit4]   2>     "hotCollections":{"collection1":500.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10002_solr":250.0},
   [junit4]   2>     "_enqueue_time_":532335566770699,
   [junit4]   2>     "requestedOps":[
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  
\"first\":\"collection1\",\n  \"second\":\"shard1\"}]}}",
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  
\"first\":\"collection1\",\n  \"second\":\"shard1\"}]}}",
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  
\"first\":\"collection1\",\n  \"second\":\"shard1\"}]}}"]}} with context 
properties: {BEFORE_ACTION=[compute]}
   [junit4]   2> 19056 INFO  (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.a.ComputePlanAction Computed Plan: 
action=ADDREPLICA&collection=collection1&shard=shard1&node=127.0.0.1:10002_solr&type=NRT
   [junit4]   2> 19061 INFO  (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.a.ComputePlanAction Computed Plan: 
action=ADDREPLICA&collection=collection1&shard=shard1&node=127.0.0.1:10002_solr&type=NRT
   [junit4]   2> 19061 INFO  (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.a.ComputePlanAction Computed Plan: 
action=ADDREPLICA&collection=collection1&shard=shard1&node=127.0.0.1:10001_solr&type=NRT
   [junit4]   2> 19061 INFO  (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 532339774 
sessionWrapper.createTime 532338824360749, this.sessionWrapper.createTime 
532338824360749 
   [junit4]   2> 19062 DEBUG (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.a.ExecutePlanAction -- processing event: {
   [junit4]   2>   "id":"1e4274deadcc3Tckrf5gmjxyt4s1w0wtuzb514x",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":532332438805699,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "hotReplicas":[
   [junit4]   2>       "{\"core_node2\":{\n    
\"core\":\"collection1_shard1_replica_n2\",\n    
\"SEARCHER.searcher.maxDoc\":0,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    
\"INDEX.sizeInBytes\":10240,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10001_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250,\n    
\"INDEX.sizeInGB\":9.5367431640625E-6,\n    \"SEARCHER.searcher.numDocs\":0,\n  
  \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node1\":{\n    
\"core\":\"collection1_shard1_replica_n1\",\n    
\"SEARCHER.searcher.maxDoc\":0,\n    \"SEARCHER.searcher.deletedDocs\":0,\n    
\"INDEX.sizeInBytes\":10240,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10002_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250,\n    
\"INDEX.sizeInGB\":9.5367431640625E-6,\n    \"SEARCHER.searcher.numDocs\":0,\n  
  \"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "__start__":3,
   [junit4]   2>     "coldNodes":{},
   [junit4]   2>     "coldShards":{},
   [junit4]   2>     "coldReplicas":[],
   [junit4]   2>     "coldCollections":{},
   [junit4]   2>     "hotShards":{"collection1":{"shard1":500.0}},
   [junit4]   2>     "hotCollections":{"collection1":500.0},
   [junit4]   2>     "hotNodes":{
   [junit4]   2>       "127.0.0.1:10001_solr":250.0,
   [junit4]   2>       "127.0.0.1:10002_solr":250.0},
   [junit4]   2>     "_enqueue_time_":532335566770699,
   [junit4]   2>     "requestedOps":[
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  
\"first\":\"collection1\",\n  \"second\":\"shard1\"}]}}",
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  
\"first\":\"collection1\",\n  \"second\":\"shard1\"}]}}",
   [junit4]   2>       "Op{action=ADDREPLICA, hints={COLL_SHARD=[{\n  
\"first\":\"collection1\",\n  \"second\":\"shard1\"}]}}"]}} with context 
properties: 
{operations=[org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica@4531d1,
 org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica@e0b264, 
org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica@a0d27a], 
AFTER_ACTION=[compute], BEFORE_ACTION=[compute, execute]}
   [junit4]   2> 19063 DEBUG (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.a.ExecutePlanAction Executing operation: 
action=ADDREPLICA&collection=collection1&shard=shard1&node=127.0.0.1:10002_solr&type=NRT
   [junit4]   2> 19101 INFO  (simCloudManagerPool-10-thread-11) [    ] 
o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10002_solr for creating new 
replica of shard shard1
   [junit4]   2> 19120 DEBUG (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.a.ExecutePlanAction Executing operation: 
action=ADDREPLICA&collection=collection1&shard=shard1&node=127.0.0.1:10002_solr&type=NRT
   [junit4]   2> 19120 INFO  (simCloudManagerPool-10-thread-15) [    ] 
o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10002_solr for creating new 
replica of shard shard1
   [junit4]   2> 19128 DEBUG (simCloudManagerPool-10-thread-12) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for collection1 / 
shard1: {"core_node1":{
   [junit4]   2>     "core":"collection1_shard1_replica_n1",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>     "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>     "INDEX.sizeInBytes":10240,
   [junit4]   2>     "node_name":"127.0.0.1:10002_solr",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "QUERY./select.requestTimes:1minRate":250,
   [junit4]   2>     "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>     "SEARCHER.searcher.numDocs":0,
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"collection1"}}
   [junit4]   2> 19137 DEBUG (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.a.ExecutePlanAction Executing operation: 
action=ADDREPLICA&collection=collection1&shard=shard1&node=127.0.0.1:10001_solr&type=NRT
   [junit4]   2> 19141 INFO  (simCloudManagerPool-10-thread-19) [    ] 
o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:10001_solr for creating new 
replica of shard shard1
   [junit4]   2> 19188 DEBUG (simCloudManagerPool-10-thread-23) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- no leader in shard1:{
   [junit4]   2>   "replicas":{"core_node2":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10001_solr",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 19188 DEBUG (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 19189 DEBUG (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: search_rate_trigger after 100ms
   [junit4]   2> 19189 DEBUG (AutoscalingActionExecutor-12-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 8634 ms for event 
id=1e4274deadcc3Tckrf5gmjxyt4s1w0wtuzb514x
   [junit4]   2> 19276 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.SolrTestCaseJ4 ###Ending testSearchRate
   [junit4]   2> 19279 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[28916BF491997653]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:         2
   [junit4]   2> ## Empty nodes:        0
   [junit4]   2> ## Dead nodes:         0
   [junit4]   2> ## Collections:        [collection1, .system]
   [junit4]   2> ## Max replicas per node:      3
   [junit4]   2> ## Min replicas per node:      3
   [junit4]   2> ## Total replicas:             6
   [junit4]   2> ## * .system           1
   [junit4]   2> ##             - active           1
   [junit4]   2> ## * collection1               5
   [junit4]   2> ##             - active           5
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##             - ADDREPLICA         3
   [junit4]   2> ##             - CREATE             1
   [junit4]   2> ##             - DELETESTATUS       3
   [junit4]   2> ##             - REQUESTSTATUS      3
   [junit4]   2> ##             - autoscaling        3
   [junit4]   2> ##             - update             2
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: search_rate_trigger
   [junit4]   2> ##             - STARTED         1
   [junit4]   2> ##             - SUCCEEDED       1
   [junit4]   2> 
   [junit4] OK      0.65s J0 | TestTriggerIntegration.testSearchRate
   [junit4]   2> 19297 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
   [junit4]   2> 19297 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 10
   [junit4]   2> 19297 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 19297 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool 
executor now
   [junit4]   2> 19297 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 19297 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 19297 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 19297 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread 
pool executor
   [junit4]   2> 19324 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 19324 DEBUG 
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 19338 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 19338 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 11
   [junit4]   2> 19339 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion -1
   [junit4]   2> 19339 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
   [junit4]   2> 19340 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 19340 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 19341 DEBUG (ScheduledTrigger-14-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 19340 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
   [junit4]   2> 19401 DEBUG (ScheduledTrigger-14-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 19424 DEBUG (ScheduledTrigger-14-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 19438 DEBUG (simCloudManagerPool-13-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 19438 DEBUG (simCloudManagerPool-13-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
12
   [junit4]   2> 19458 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
   [junit4]   2> 19536 DEBUG (ScheduledTrigger-14-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 19536 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 19536 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
   [junit4]   2> 19539 DEBUG (simCloudManagerPool-13-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 19540 DEBUG (simCloudManagerPool-13-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
13
   [junit4]   2> 19541 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
   [junit4]   2> 19541 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 19541 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 19542 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 
127.0.0.1:10003_solr
   [junit4]   2> 19542 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 19542 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
   [junit4]   2> 19542 DEBUG (ScheduledTrigger-14-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> 19558 DEBUG (ScheduledTrigger-14-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 19573 DEBUG (ScheduledTrigger-14-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> 19573 DEBUG (simCloudManagerPool-13-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 19574 DEBUG (simCloudManagerPool-13-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
14
   [junit4]   2> 19575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 14
   [junit4]   2> 19575 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 19575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger - Initial 
livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10002_solr]
   [junit4]   2> 19575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 19575 DEBUG (ScheduledTrigger-14-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> 19575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
   [junit4]   2> 19575 DEBUG (ScheduledTrigger-14-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost 
nodes: [127.0.0.1:10003_solr]
   [junit4]   2> 19575 DEBUG (ScheduledTrigger-14-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"1e42e99469c2dTckrf5gmjxyt4s1w0wtuzb5150",
   [junit4]   2>   "source":"node_lost_restore_trigger",
   [junit4]   2>   "eventTime":532363767880749,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[532363767880749],
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 19575 DEBUG (ScheduledTrigger-14-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_lost_restore_trigger]
   [junit4]   2> 19605 DEBUG (simCloudManagerPool-13-thread-4) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- no leader in shard1:{
   [junit4]   2>   "replicas":{"core_node3":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10002_solr",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 19606 DEBUG (AutoscalingActionExecutor-15-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"1e42e99469c2dTckrf5gmjxyt4s1w0wtuzb5150",
   [junit4]   2>   "source":"node_lost_restore_trigger",
   [junit4]   2>   "eventTime":532363767880749,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[532363767880749],
   [junit4]   2>     "_enqueue_time_":532365488725749,
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 19607 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
   [junit4]   2> 19607 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:         2
   [junit4]   2> ## Empty nodes:        1
   [junit4]   2> ## Dead nodes:         1
   [junit4]   2> ##             127.0.0.1:10003_solr
   [junit4]   2> ## Collections:        [.system]
   [junit4]   2> ## Max replicas per node:      1
   [junit4]   2> ## Min replicas per node:      0
   [junit4]   2> ## Total replicas:             1
   [junit4]   2> ## * .system           1
   [junit4]   2> ##             - active           1
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##             - autoscaling        3
   [junit4]   2> ##             - update             1
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> ## * Trigger: node_lost_restore_trigger
   [junit4]   2> ##             - STARTED         1
   [junit4]   2> 
   [junit4] OK      0.33s J0 | 
TestTriggerIntegration.testNodeLostTriggerRestoreState
   [junit4]   2> 19612 DEBUG (simCloudManagerPool-13-thread-7) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- no leader in shard1:{
   [junit4]   2>   "replicas":{"core_node3":{
   [junit4]   2>       "core":".system_shard1_replica_n1",
   [junit4]   2>       "SEARCHER.searcher.maxDoc":0,
   [junit4]   2>       "SEARCHER.searcher.deletedDocs":0,
   [junit4]   2>       "INDEX.sizeInBytes":10240,
   [junit4]   2>       "node_name":"127.0.0.1:10002_solr",
   [junit4]   2>       "state":"active",
   [junit4]   2>       "type":"NRT",
   [junit4]   2>       "INDEX.sizeInGB":9.5367431640625E-6,
   [junit4]   2>       "SEARCHER.searcher.numDocs":0}},
   [junit4]   2>   "range":"80000000-7fffffff",
   [junit4]   2>   "state":"active"}
   [junit4]   2> 19621 INFO  
(TEST-TestTriggerIntegration.testEventQueue-seed#[28916BF491997653]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testEventQueue
   [junit4]   2> 19623 DEBUG 
(TEST-TestTriggerIntegration.testEventQueue-seed#[28916BF491997653]) [    ] 
o.a.s.c.a.OverseerTrig

[...truncated too long message...]

er Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
   [junit4]   2> 30969 DEBUG (ScheduledTrigger-149-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 30969 DEBUG (ScheduledTrigger-149-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 30989 DEBUG (ScheduledTrigger-149-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 30989 DEBUG (ScheduledTrigger-149-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 30989 DEBUG (ScheduledTrigger-149-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 30998 DEBUG (simCloudManagerPool-148-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 30998 DEBUG (simCloudManagerPool-148-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
56
   [junit4]   2> 30998 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 56
   [junit4]   2> 30999 INFO  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 30999 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger - 
Initial livenodes: [127.0.0.1:10045_solr, 127.0.0.1:10041_solr, 
127.0.0.1:10046_solr]
   [junit4]   2> 30999 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger 
instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 30999 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 56, lastZnodeVersion 56
   [junit4]   2> 30999 DEBUG (ScheduledTrigger-149-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 30999 DEBUG (ScheduledTrigger-149-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 31010 DEBUG (ScheduledTrigger-149-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 31020 DEBUG (ScheduledTrigger-149-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 31020 DEBUG (ScheduledTrigger-149-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 31031 DEBUG (ScheduledTrigger-149-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 31042 DEBUG (ScheduledTrigger-149-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 31042 DEBUG (ScheduledTrigger-149-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 31051 DEBUG (ScheduledTrigger-149-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 31065 DEBUG (ScheduledTrigger-149-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 31117 DEBUG (ScheduledTrigger-149-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 31117 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTriggerRestoreState
   [junit4]   2> 31117 DEBUG (ScheduledTrigger-149-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 31117 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[28916BF491997653])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
   [junit4]   2> ############ CLUSTER STATE ############
   [junit4]   2> #######################################
   [junit4]   2> ## Live nodes:         3
   [junit4]   2> ## Empty nodes:        3
   [junit4]   2> ## Dead nodes:         0
   [junit4]   2> ## Collections:        []
   [junit4]   2> ## Max replicas per node:      0
   [junit4]   2> ## Min replicas per node:      0
   [junit4]   2> ## Total replicas:             0
   [junit4]   2> ######### Solr op counts ##########
   [junit4]   2> ##             - autoscaling        3
   [junit4]   2> ######### Autoscaling event counts ###########
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestTriggerIntegration 
-Dtests.method=testNodeAddedTriggerRestoreState -Dtests.seed=28916BF491997653 
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ar -Dtests.timezone=PST 
-Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.27s J0 | 
TestTriggerIntegration.testNodeAddedTriggerRestoreState <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: The trigger did not 
fire at all
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([28916BF491997653:A0ACE28BAB5997FE]:0)
   [junit4]    >        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeAddedTriggerRestoreState(TestTriggerIntegration.java:383)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 31118 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 31118 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 31118 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 31118 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool 
executor
   [junit4]   2> 31118 WARN  (Simulated OverseerAutoScalingTriggerThread) [    
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 31118 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 31118 DEBUG 
(SUITE-TestTriggerIntegration-seed#[28916BF491997653]-worker) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> NOTE: leaving temporary files on disk at: 
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.autoscaling.sim.TestTriggerIntegration_28916BF491997653-002
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
docValues:{}, maxPointsInLeafNode=1505, maxMBSortInHeap=6.6407289666570914, 
sim=RandomSimilarity(queryNorm=false): {}, locale=ar, timezone=PST
   [junit4]   2> NOTE: Windows 10 10.0 x86/Oracle Corporation 1.8.0_172 
(32-bit)/cpus=3,threads=1,free=19774264,total=33554432
   [junit4]   2> NOTE: All tests run in this JVM: [TestTriggerIntegration, 
TestTriggerIntegration, TestTriggerIntegration]
   [junit4] Completed [5/5 (3!)] on J0 in 4.00s, 11 tests, 1 failure <<< 
FAILURES!

[...truncated 12 lines...]
BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\common-build.xml:1568:
 The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\common-build.xml:1092:
 There were test failures: 5 suites, 55 tests, 3 failures [seed: 
28916BF491997653]

Total time: 35 seconds

[repro] Setting last failure code to 1

[repro] Failures:
[repro]   3/5 failed: 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
[repro] Exiting with code 1

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST lucene\build move 
/y lucene\build lucene\build.repro 
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST solr\build move /y 
solr\build solr\build.repro 
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST lucene\build.orig 
move /y lucene\build.orig lucene\build 
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST solr\build.orig 
move /y solr\build.orig solr\build 
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>exit 0 
Archiving artifacts
Setting 
ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting 
ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting 
ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all 
warnings.
<Git Blamer> GIT_COMMIT=5a7cb240652756bcd4d28231797e053e8bb40ec7, 
workspace=C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows
[WARNINGS] Computing warning deltas based on reference build #740
Recording test results
Setting 
ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting 
ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting 
ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting 
ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting 
ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to