Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.0-Linux/231/
Java: 32bit/jdk1.8.0_172 -server -XX:+UseSerialGC
1 tests failed.
FAILED:
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeAddedTriggerRestoreState
Error Message:
OverseerTriggerThread never caught up to the latest znodeVersion
Stack Trace:
java.util.concurrent.TimeoutException: OverseerTriggerThread never caught up to
the latest znodeVersion
at
__randomizedtesting.SeedInfo.seed([A27319673644C5A9:2A4E90180C842404]:0)
at org.apache.solr.util.TimeOut.waitFor(TimeOut.java:66)
at
org.apache.solr.cloud.autoscaling.sim.SimSolrCloudTestCase.assertAutoscalingUpdateComplete(SimSolrCloudTestCase.java:71)
at
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeAddedTriggerRestoreState(TestSimTriggerIntegration.java:456)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 14573 lines...]
[junit4] Suite:
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
[junit4] 2> Creating dataDir:
/home/jenkins/workspace/Lucene-Solr-8.0-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_A27319673644C5A9-001/init-core-data-001
[junit4] 2> 1685525 INFO
(SUITE-TestSimTriggerIntegration-seed#[A27319673644C5A9]-worker) [ ]
o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false)
w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1685526 INFO
(SUITE-TestSimTriggerIntegration-seed#[A27319673644C5A9]-worker) [ ]
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via:
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 1685526 INFO
(SUITE-TestSimTriggerIntegration-seed#[A27319673644C5A9]-worker) [ ]
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks:
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1685529 INFO
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
[junit4] 2> 1685531 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader:
127.0.0.1:10248_solr
[junit4] 2> 1685531 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and
.scheduled_maintenance triggers
[junit4] 2> 1685531 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states,
currentVersion=0
[junit4] 2> 1685531 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode
version 1
[junit4] 2> 1685531 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
[junit4] 2> 1685531 INFO
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics
history in memory.
[junit4] 2> 1685532 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
[junit4] 2> 1685532 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 1685532 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial
livenodes: [127.0.0.1:10249_solr, 127.0.0.1:10248_solr]
[junit4] 2> 1685532 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1685532 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 1685532 DEBUG (ScheduledTrigger-7756-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1685532 DEBUG (simCloudManagerPool-7755-thread-2) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1685532 DEBUG (simCloudManagerPool-7755-thread-2) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
2
[junit4] 2> 1685533 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 1685533 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1685533 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 1685534 DEBUG (simCloudManagerPool-7755-thread-3) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1685534 DEBUG (simCloudManagerPool-7755-thread-3) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
3
[junit4] 2> 1685535 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 1685535 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
[junit4] 2> 1685535 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger -
Initial livenodes: [127.0.0.1:10249_solr, 127.0.0.1:10248_solr]
[junit4] 2> 1685535 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1685535 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 1685535 DEBUG (ScheduledTrigger-7756-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1685553 DEBUG (ScheduledTrigger-7756-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 2
[junit4] 2> 1685556 DEBUG (ScheduledTrigger-7756-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 3 and last live nodes: 2
[junit4] 2> 1685556 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.s.SimClusterStateProvider -- created marker:
/autoscaling/nodeLost/127.0.0.1:10250_solr
[junit4] 2> 1685573 DEBUG (ScheduledTrigger-7756-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 3
[junit4] 2> 1685573 DEBUG (ScheduledTrigger-7756-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10250_solr
[junit4] 2> 1685576 DEBUG (ScheduledTrigger-7756-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 3
[junit4] 2> 1685576 DEBUG (ScheduledTrigger-7756-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10250_solr
[junit4] 2> 1685578 DEBUG (simCloudManagerPool-7755-thread-4) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1685579 DEBUG (simCloudManagerPool-7755-thread-4) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
4
[junit4] 2> 1685579 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 1685579 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
[junit4] 2> 1685579 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_restore_trigger -
Initial livenodes: [127.0.0.1:10249_solr, 127.0.0.1:10248_solr]
[junit4] 2> 1685579 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1685579 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 1685579 DEBUG (ScheduledTrigger-7756-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1685593 DEBUG (ScheduledTrigger-7756-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1685600 DEBUG (ScheduledTrigger-7756-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger
with currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1685600 DEBUG (ScheduledTrigger-7756-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost
nodes: [127.0.0.1:10250_solr]
[junit4] 2> 1685600 DEBUG (ScheduledTrigger-7756-thread-4) [ ]
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"3ef1793dbd11dT9d2mneax2y2o7qtzdnfujzeb2",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":1107309474074909,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1107309474074909],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "nodeNames":["127.0.0.1:10250_solr"]}}
[junit4] 2> 1685600 DEBUG (ScheduledTrigger-7756-thread-4) [ ]
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas,
node_lost_restore_trigger]
[junit4] 2> 1685600 DEBUG (ScheduledTrigger-7756-thread-4) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"3ef1793dbd11dT9d2mneax2y2o7qtzdnfujzeb2",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":1107309474074909,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1107309474074909],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1107310676705559,
[junit4] 2> "nodeNames":["127.0.0.1:10250_solr"]}}
[junit4] 2> 1685600 DEBUG (AutoscalingActionExecutor-7757-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"3ef1793dbd11dT9d2mneax2y2o7qtzdnfujzeb2",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":1107309474074909,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1107309474074909],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1107310676705559,
[junit4] 2> "nodeNames":["127.0.0.1:10250_solr"]}}
[junit4] 2> 1685601 DEBUG (AutoscalingActionExecutor-7757-thread-1) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"3ef1793dbd11dT9d2mneax2y2o7qtzdnfujzeb2",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":1107309474074909,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1107309474074909],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1107310676705559,
[junit4] 2> "nodeNames":["127.0.0.1:10250_solr"]}}
[junit4] 2> 1685601 DEBUG (AutoscalingActionExecutor-7757-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 1685601 DEBUG (AutoscalingActionExecutor-7757-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_restore_trigger after
100ms
[junit4] 2> 1685601 DEBUG (AutoscalingActionExecutor-7757-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing took 16 ms for event
id=3ef1793dbd11dT9d2mneax2y2o7qtzdnfujzeb2
[junit4] 2> 1685601 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool
executor now
[junit4] 2> 1685601 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are
closed, exiting.
[junit4] 2> 1685601 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 1685602 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 1685602 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread
pool executor
[junit4] 2> 1685602 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 1685602 DEBUG
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed
explicitly
[junit4] 2> 1685603 INFO
(TEST-TestSimTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A27319673644C5A9])
[ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
[junit4] 2> 1685606 INFO
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[A27319673644C5A9]) [
] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
[junit4] 2> 1685608 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[A27319673644C5A9]) [
] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader:
127.0.0.1:10251_solr
[junit4] 2> 1685609 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and
.scheduled_maintenance triggers
[junit4] 2> 1685609 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode
version 1
[junit4] 2> 1685609 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[A27319673644C5A9]) [
] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states,
currentVersion=0
[junit4] 2> 1685609 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[A27319673644C5A9]) [
] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 0
[junit4] 2> 1685609 INFO
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[A27319673644C5A9]) [
] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics
history in memory.
[junit4] 2> 1685609 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
[junit4] 2> 1685609 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 1685609 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial
livenodes: [127.0.0.1:10251_solr, 127.0.0.1:10252_solr]
[junit4] 2> 1685609 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1685609 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 1685609 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 2 and last live nodes: 2
[junit4] 2> 1685610 DEBUG (simCloudManagerPool-7761-thread-2) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1685610 DEBUG (simCloudManagerPool-7761-thread-2) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
2
[junit4] 2> 1685610 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 1685611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1685611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 1685611 DEBUG (simCloudManagerPool-7761-thread-3) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1685611 DEBUG (simCloudManagerPool-7761-thread-3) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
3
[junit4] 2> 1685611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 1685611 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.s.TestSimTriggerIntegration TestTriggerAction init
[junit4] 2> 1685611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial
livenodes: [127.0.0.1:10251_solr, 127.0.0.1:10252_solr]
[junit4] 2> 1685611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test,
class=org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration$TestTriggerAction}],
enabled=true}
[junit4] 2> 1685611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1685611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 1685611 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685611 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1685616 DEBUG
(TEST-TestSimTriggerIntegration.testNodeAddedTrigger-seed#[A27319673644C5A9]) [
] o.a.s.c.a.s.SimClusterStateProvider -- creating marker:
/autoscaling/nodeAdded/127.0.0.1:10253_solr
[junit4] 2> 1685630 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 2
[junit4] 2> 1685632 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685632 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685632 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10253_solr at time
1107312270682859
[junit4] 2> 1685651 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685652 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685652 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685652 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing
registered processor for nodes: [127.0.0.1:10253_solr] added at times
[1107312270682859], now=1107313278324009
[junit4] 2> 1685652 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"3ef183a8caaebT9d2mneax2y2o7qtzdnfujzeb4",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1107312270682859,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1107312270682859],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "nodeNames":["127.0.0.1:10253_solr"]}}
[junit4] 2> 1685653 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas,
node_added_trigger]
[junit4] 2> 1685653 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"3ef183a8caaebT9d2mneax2y2o7qtzdnfujzeb4",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1107312270682859,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1107312270682859],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1107313304101809,
[junit4] 2> "nodeNames":["127.0.0.1:10253_solr"]}}
[junit4] 2> 1685653 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10253_solr
[junit4] 2> 1685653 DEBUG (AutoscalingActionExecutor-7763-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"3ef183a8caaebT9d2mneax2y2o7qtzdnfujzeb4",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1107312270682859,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1107312270682859],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1107313304101809,
[junit4] 2> "nodeNames":["127.0.0.1:10253_solr"]}}
[junit4] 2> 1685653 DEBUG (AutoscalingActionExecutor-7763-thread-1) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"3ef183a8caaebT9d2mneax2y2o7qtzdnfujzeb4",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1107312270682859,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1107312270682859],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1107313304101809,
[junit4] 2> "nodeNames":["127.0.0.1:10253_solr"]}}
[junit4] 2> 1685653 DEBUG (AutoscalingActionExecutor-7763-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 1685653 DEBUG (AutoscalingActionExecutor-7763-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 1685653 DEBUG (AutoscalingActionExecutor-7763-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing took 8 ms for event
id=3ef183a8caaebT9d2mneax2y2o7qtzdnfujzeb4
[junit4] 2> 1685654 DEBUG (simCloudManagerPool-7761-thread-4) [ ]
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1685654 DEBUG (simCloudManagerPool-7761-thread-4) [ ]
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version
4
[junit4] 2> 1685654 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 1685654 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread -- cleaning old nodeLost / nodeAdded markers
[junit4] 2> 1685654 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 1685753 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685753 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685753 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685774 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685774 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685774 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685794 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685794 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685794 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685814 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685814 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685814 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685834 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685834 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685834 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685854 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685854 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685854 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685874 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685874 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685874 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685895 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685895 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685895 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685915 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685915 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685915 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685935 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685935 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685935 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685955 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685955 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685955 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685975 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685975 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685975 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1685995 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1685995 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1685995 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686015 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686015 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686016 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686036 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686036 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686036 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686056 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686056 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686056 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686076 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686076 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686076 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686096 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686096 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686096 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686116 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686116 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686116 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686136 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686136 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686137 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686157 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686157 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686157 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686177 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686177 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686177 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686197 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686197 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686197 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686217 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686217 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686217 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686237 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686237 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686237 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686257 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686257 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686257 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686278 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686278 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686278 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686298 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686298 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686298 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686318 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686318 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686318 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686338 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686338 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686338 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686358 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686358 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686358 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686378 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686378 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686378 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686398 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686399 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686399 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686419 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686419 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686419 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686439 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686439 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686439 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686459 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686459 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686459 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686479 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686479 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686479 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686499 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686499 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686499 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686519 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686519 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686519 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686540 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686540 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686540 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686560 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686560 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686560 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686580 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686580 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686580 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686600 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686600 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686600 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686620 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686620 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686620 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686640 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686640 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686640 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686661 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686661 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686661 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686681 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686681 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686681 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686701 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686701 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686701 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686721 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686721 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686721 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686741 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686741 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686741 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686761 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686761 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686761 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686782 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686782 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686782 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686802 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686802 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686802 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686822 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686822 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686822 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686842 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686842 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686842 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686862 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686862 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686863 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686883 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686883 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686883 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686903 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686903 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686903 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686923 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686923 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686923 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686943 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686943 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686943 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686963 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686963 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686963 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1686984 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1686984 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1686984 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687004 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687004 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687004 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687024 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687024 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687024 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687044 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687044 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687044 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687064 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687064 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687064 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687084 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687084 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687084 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687105 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687105 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687105 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687125 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687125 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687125 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687145 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687145 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687145 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687165 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687165 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687165 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687185 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687185 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687185 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687205 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687205 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687205 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687226 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687226 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687226 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687246 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687246 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687246 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687266 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687266 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687266 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687286 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687286 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687286 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687306 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687306 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687306 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687327 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687327 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687327 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687347 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687347 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687347 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687367 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687367 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687367 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687387 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687387 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687387 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687407 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687407 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687407 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687427 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687427 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687427 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687448 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687448 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687448 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687468 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687468 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687468 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687488 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687488 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687488 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687508 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687508 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687508 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687528 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687528 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687528 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687548 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687548 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687548 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687569 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687569 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687569 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687589 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687589 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687589 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687609 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687609 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687609 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687629 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687629 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687629 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687649 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687649 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687649 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687669 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687670 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687670 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687690 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687690 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687690 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687710 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687710 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687710 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687730 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687730 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687730 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687750 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687750 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687750 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687770 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687770 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687770 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687791 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687791 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687791 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687811 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687811 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687811 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687831 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687831 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687831 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687851 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687851 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687851 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687871 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687871 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687871 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687891 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687891 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687892 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687912 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687912 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687912 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687932 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687932 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687932 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687952 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687952 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687952 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687972 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687972 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687972 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1687992 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1687992 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1687992 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688013 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688013 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688013 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688033 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688033 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688033 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688053 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688053 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688053 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688073 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688073 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688073 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688093 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688093 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688093 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688113 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688113 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688113 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688134 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688134 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688134 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688154 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688154 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688154 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688174 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688174 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688174 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688194 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688194 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688194 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688214 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688214 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688214 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688234 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688234 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688234 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688255 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688255 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688255 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688275 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688275 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688275 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688295 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688295 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688295 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688315 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688315 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688315 DEBUG (ScheduledTrigger-7762-thread-1) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688335 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688335 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688335 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688355 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688355 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688355 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688376 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688376 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688376 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688396 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688396 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688396 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688416 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688416 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688416 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688436 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688436 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688436 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688456 DEBUG (ScheduledTrigger-7762-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688456 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688456 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688476 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1688477 DEBUG (ScheduledTrigger-7762-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 3 and last live nodes: 3
[junit4] 2> 1688477 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1688497 DEBUG (ScheduledTrigger-7762-thread-4) [ ]
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_ad
[...truncated too long message...]
5268059],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1107803684712659,
[junit4] 2> "nodeNames":["127.0.0.1:10282_solr"]}}
[junit4] 2> 1695461 DEBUG (ScheduledTrigger-7830-thread-3) [ ]
o.a.s.c.a.NodeAddedTrigger Removing new node from tracking: 127.0.0.1:10282_solr
[junit4] 2> 1695461 DEBUG (AutoscalingActionExecutor-7831-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"3ef8842ca2a5bT9d2mneax2y2o7qtzdnfujzebu",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1107793445268059,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1107793445268059],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1107803684712659,
[junit4] 2> "nodeNames":["127.0.0.1:10282_solr"]}}
[junit4] 2> 1695461 DEBUG (AutoscalingActionExecutor-7831-thread-1) [ ]
o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event {
[junit4] 2> "id":"3ef8842ca2a5bT9d2mneax2y2o7qtzdnfujzebu",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1107793445268059,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1107793445268059],
[junit4] 2> "preferredOperation":"movereplica",
[junit4] 2> "_enqueue_time_":1107803684712659,
[junit4] 2> "nodeNames":["127.0.0.1:10282_solr"]}}
[junit4] 2> 1695461 DEBUG (AutoscalingActionExecutor-7831-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 1695461 DEBUG (AutoscalingActionExecutor-7831-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 1695461 DEBUG (AutoscalingActionExecutor-7831-thread-1) [ ]
o.a.s.c.a.ScheduledTriggers -- processing took 10 ms for event
id=3ef8842ca2a5bT9d2mneax2y2o7qtzdnfujzebu
[junit4] 2> 1695461 DEBUG
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool
executor now
[junit4] 2> 1695461 INFO (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are
closed, exiting.
[junit4] 2> 1695462 DEBUG
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 1695462 DEBUG
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 1695462 DEBUG
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread
pool executor
[junit4] 2> 1695462 DEBUG
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 1695462 DEBUG
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A27319673644C5A9])
[ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed
explicitly
[junit4] 2> 1695462 INFO
(TEST-TestSimTriggerIntegration.testEventFromRestoredState-seed#[A27319673644C5A9])
[ ] o.a.s.SolrTestCaseJ4 ###Ending testEventFromRestoredState
[junit4] 2> NOTE: leaving temporary files on disk at:
/home/jenkins/workspace/Lucene-Solr-8.0-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimTriggerIntegration_A27319673644C5A9-001
[junit4] 2> NOTE: test params are: codec=Lucene80,
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1e32143),
locale=es-CU, timezone=Europe/Riga
[junit4] 2> NOTE: Linux 4.18.0-15-generic i386/Oracle Corporation
1.8.0_172 (32-bit)/cpus=8,threads=1,free=249449144,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [JsonLoaderTest,
HdfsBasicDistributedZkTest, TestUtilizeNode, TestCoreBackup,
SuggestComponentContextFilterQueryTest, ReturnFieldsTest,
TestReplicaProperties, CacheHeaderTest, TestFieldCacheReopen,
TestNumericTokenStream, IndexSchemaTest, TestConfigsApi,
TestAnalyzedSuggestions, SpatialFilterTest, TestNonDefinedSimilarityFactory,
TestLuceneMatchVersion, ShufflingReplicaListTransformerTest,
MetricsHistoryHandlerTest, CollectionTooManyReplicasTest,
AdminHandlersProxyTest, TaggerTest, TestIntervalFaceting,
TestNestedUpdateProcessor, PKIAuthenticationIntegrationTest,
TermVectorComponentTest, TestHashPartitioner, ConjunctionSolrSpellCheckerTest,
RegexBoostProcessorTest, SegmentsInfoRequestHandlerTest,
ClassificationUpdateProcessorFactoryTest, TestManagedSchemaThreadSafety,
TestMinMaxOnMultiValuedField, TestCollectionsAPIViaSolrCloudCluster,
DistributedFacetExistsSmallTest, CheckHdfsIndexTest,
RecoveryAfterSoftCommitTest, ZkStateWriterTest, TestTrieFacet,
SpellPossibilityIteratorTest, TestJsonFacetRefinement, LeaderElectionTest,
BasicDistributedZk2Test, RulesTest, SolrIndexConfigTest, TestRandomFaceting,
TestUnInvertedFieldException, TestTlogReplica, TestConfigSetsAPI,
TestDistribIDF, TestUtils, TestBinaryResponseWriter,
TestEmbeddedSolrServerConstructors, BigEndianAscendingWordSerializerTest,
TestSQLHandler, SaslZkACLProviderTest, TestSubQueryTransformerCrossCore,
TestSolrCloudSnapshots, CursorMarkTest, CdcrReplicationHandlerTest,
ZkFailoverTest, TestCustomSort, TestXmlQParserPlugin,
TestLegacyNumericRangeQueryBuilder, AddBlockUpdateTest, TestMacros,
OutOfBoxZkACLAndCredentialsProvidersTest, DocValuesMissingTest, TestSearchPerf,
ConnectionReuseTest, TestCloudRecovery2, FullHLLTest,
TestGraphTermsQParserPlugin, TestFieldCollectionResource,
TestHighlightDedupGrouping, RootFieldTest, NoCacheHeaderTest,
TestPHPSerializedResponseWriter, ChaosMonkeyNothingIsSafeTest,
TestInfoStreamLogging, DistanceFunctionTest,
TestOverriddenPrefixQueryForCustomFieldType, TestTolerantSearch,
CloudExitableDirectoryReaderTest, ConcurrentCreateRoutedAliasTest,
TestExtendedDismaxParser, HdfsTlogReplayBufferedWhileIndexingTest,
TestFreeTextSuggestions, SolrCoreCheckLockOnStartupTest,
TestShortCircuitedRequests, TestSimLargeCluster, QueryEqualityTest,
TestSchemaSimilarityResource, TestDocTermOrds, TestBadConfig,
SolrRrdBackendFactoryTest, BlockJoinFacetRandomTest, BooleanFieldTest,
UUIDUpdateProcessorFallbackTest, RequestLoggingTest,
PreAnalyzedFieldManagedSchemaCloudTest, TestAtomicUpdateErrorCases,
FastVectorHighlighterTest, CreateCollectionCleanupTest,
PhrasesIdentificationComponentTest, AnalyticsMergeStrategyTest,
SpatialHeatmapFacetsTest, UpdateLogTest, SolrGraphiteReporterTest,
CSVRequestHandlerTest, ConcurrentDeleteAndCreateCollectionTest,
CoreMergeIndexesAdminHandlerTest, ReplaceNodeNoTargetTest, SyncSliceTest,
SparseHLLTest, TestManagedSynonymGraphFilterFactory, TokenizerChainTest,
TestSchemaManager, PeerSyncWithIndexFingerprintCachingTest,
ExitableDirectoryReaderTest, OrderedExecutorTest,
ChaosMonkeySafeLeaderWithPullReplicasTest,
DistributedFacetPivotSmallAdvancedTest, OverseerModifyCollectionTest,
TestSolrCloudWithDelegationTokens, ResponseLogComponentTest,
ProtectedTermFilterFactoryTest, TestFieldCacheVsDocValues, TestLockTree,
TestComplexPhraseLeadingWildcard, HttpPartitionTest,
ConfigureRecoveryStrategyTest, MaxSizeAutoCommitTest, TestSolrXml,
NodeAddedTriggerIntegrationTest, TestCursorMarkWithoutUniqueKey,
DeleteLastCustomShardedReplicaTest, AssignBackwardCompatibilityTest,
ClusterStateUpdateTest, CollectionPropsTest, DeleteShardTest, DeleteStatusTest,
DistribDocExpirationUpdateProcessorTest, LeaderFailureAfterFreshStartTest,
MissingSegmentRecoveryTest, MoveReplicaHDFSFailoverTest, MoveReplicaHDFSTest,
MoveReplicaTest, MultiSolrCloudTestCaseTest, OverseerTaskQueueTest,
RestartWhileUpdatingTest, RollingRestartTest, SliceStateTest,
SolrCloudExampleTest, TestCloudDeleteByQuery, TestCloudPivotFacet,
TestDistribDocBasedVersion, TestPrepRecovery, TestRandomRequestDistribution,
TestRequestForwarding, TestSSLRandomization, TestStressCloudBlindAtomicUpdates,
ZkShardTermsTest, SimpleCollectionCreateDeleteTest,
AutoAddReplicasPlanActionTest, AutoScalingHandlerTest, ComputePlanActionTest,
IndexSizeTriggerTest, NodeLostTriggerIntegrationTest, NodeLostTriggerTest,
ScheduledMaintenanceTriggerTest, TestPolicyCloud, TestSimComputePlanAction,
TestSimDistribStateManager, TestSimDistributedQueue, TestSimNodeLostTrigger,
TestSimTriggerIntegration]
[junit4] Completed [547/840 (1!)] on J1 in 10.01s, 11 tests, 1 error <<<
FAILURES!
[...truncated 46217 lines...]
[repro] Jenkins log URL:
https://jenkins.thetaphi.de/job/Lucene-Solr-8.0-Linux/231/consoleText
[repro] Revision: 0376bc0052a53480ecb2edea7dfe58298bda5deb
[repro] Ant options: "-Dargs=-server -XX:+UseSerialGC"
[repro] Repro line: ant test -Dtestcase=TestSimTriggerIntegration
-Dtests.method=testNodeAddedTriggerRestoreState -Dtests.seed=A27319673644C5A9
-Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es-CU
-Dtests.timezone=Europe/Riga -Dtests.asserts=true
-Dtests.file.encoding=ISO-8859-1
[repro] ant clean
[...truncated 6 lines...]
[repro] Test suites by module:
[repro] solr/core
[repro] TestSimTriggerIntegration
[repro] ant compile-test
[...truncated 3572 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5
-Dtests.class="*.TestSimTriggerIntegration" -Dtests.showOutput=onerror
"-Dargs=-server -XX:+UseSerialGC" -Dtests.seed=A27319673644C5A9
-Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es-CU
-Dtests.timezone=Europe/Riga -Dtests.asserts=true
-Dtests.file.encoding=ISO-8859-1
[...truncated 120 lines...]
[repro] Failures:
[repro] 0/5 failed:
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration
[repro] Exiting with code 0
[...truncated 43 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
