Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/8862/ Java: 64bit/jdk-12.0.2 -XX:+UseCompressedOops -XX:+UseSerialGC
1 tests failed. FAILED: org.apache.solr.cloud.autoscaling.sim.TestSimScenario.testSuggestions Error Message: OverseerTriggerThread never caught up to the latest znodeVersion Stack Trace: java.util.concurrent.TimeoutException: OverseerTriggerThread never caught up to the latest znodeVersion at __randomizedtesting.SeedInfo.seed([EA19F5D72A768827:4CC5CC22E442DFD9]:0) at org.apache.solr.util.TimeOut.waitFor(TimeOut.java:66) at org.apache.solr.cloud.autoscaling.sim.SimScenario$LoadAutoscaling.execute(SimScenario.java:466) at org.apache.solr.cloud.autoscaling.sim.SimScenario.run(SimScenario.java:1110) at org.apache.solr.cloud.autoscaling.sim.TestSimScenario.testSuggestions(TestSimScenario.java:114) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:567) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370) at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826) at java.base/java.lang.Thread.run(Thread.java:835) Build Log: [...truncated 13661 lines...] [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestSimScenario [junit4] 2> 592245 INFO (SUITE-TestSimScenario-seed#[EA19F5D72A768827]-worker) [ ] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\server\solr\configsets\_default\conf' [junit4] 2> 592246 INFO (SUITE-TestSimScenario-seed#[EA19F5D72A768827]-worker) [ ] o.a.s.SolrTestCaseJ4 Created dataDir: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J2\temp\solr.cloud.autoscaling.sim.TestSimScenario_EA19F5D72A768827-001\data-dir-72-001 [junit4] 2> 592246 WARN (SUITE-TestSimScenario-seed#[EA19F5D72A768827]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1 [junit4] 2> 592246 INFO (SUITE-TestSimScenario-seed#[EA19F5D72A768827]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false [junit4] 2> 592246 INFO (SUITE-TestSimScenario-seed#[EA19F5D72A768827]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) [junit4] 2> 592246 INFO (SUITE-TestSimScenario-seed#[EA19F5D72A768827]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom [junit4] 2> 592250 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testSplitShard [junit4] 2> 592267 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 1. CreateCluster numNodes=2 [junit4] 2> 592268 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario CreateCluster numNodes=2 [junit4] 2> 592275 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10006_solr [junit4] 2> 592275 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory. [junit4] 2> 592275 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers [junit4] 2> 592275 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 2. LoadAutoscaling json={'cluster-policy':[]} [junit4] 2> 592275 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1 [junit4] 2> 592275 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario LoadAutoscaling json={'cluster-policy':[]} [junit4] 2> 592275 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0 [junit4] 2> 592275 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1 [junit4] 2> 592275 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2 [junit4] 2> 592276 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10007_solr] [junit4] 2> 592276 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded markers [junit4] 2> 592276 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 1 [junit4] 2> 592276 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2 [junit4] 2> 592276 DEBUG (ScheduledTrigger-5132-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592276 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded markers [junit4] 2> 592276 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2 [junit4] 2> 592276 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 3. RunSolrRequest path=/admin/collections&action=CREATE&name=testCollection&numShards=2&replicationFactor=2&maxShardsPerNode=5 [junit4] 2> 592276 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario RunSolrRequest path=/admin/collections&action=CREATE&name=testCollection&numShards=2&replicationFactor=2&maxShardsPerNode=5 [junit4] 2> 592277 DEBUG (simCloudManagerPool-5131-thread-1) [ ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection testCollection [junit4] 2> 592288 DEBUG (simCloudManagerPool-5131-thread-1) [ ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection testCollection [junit4] 2> 592289 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 4. WaitCollection collection=testCollection&shards=2&replicas=2 [junit4] 2> 592289 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario WaitCollection collection=testCollection&shards=2&replicas=2 [junit4] 2> 592297 DEBUG (ScheduledTrigger-5132-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592299 DEBUG (simCloudManagerPool-5131-thread-2) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard1) [junit4] 2> 592299 DEBUG (simCloudManagerPool-5131-thread-2) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard1 (currentVersion=2): {"core_node1":{ [junit4] 2> "core":"testCollection_shard1_replica_n1", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10007_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":10240, [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":9.5367431640625E-6, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 592319 DEBUG (ScheduledTrigger-5132-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592323 DEBUG (simCloudManagerPool-5131-thread-4) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard2) [junit4] 2> 592323 DEBUG (simCloudManagerPool-5131-thread-4) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard2 (currentVersion=4): {"core_node3":{ [junit4] 2> "core":"testCollection_shard2_replica_n3", [junit4] 2> "shard":"shard2", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10007_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":10240, [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":9.5367431640625E-6, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 592334 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 5. SetShardMetrics collection=testCollection&shard=shard1&INDEX.sizeInBytes=1000000000 [junit4] 2> 592334 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario SetShardMetrics collection=testCollection&shard=shard1&INDEX.sizeInBytes=1000000000 [junit4] 2> 592336 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 6. SetNodeMetrics nodeset=#ANY&freedisk=1.5 [junit4] 2> 592336 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario SetNodeMetrics nodeset=#ANY&freedisk=1.5 [junit4] 2> 592336 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 7. RunSolrRequest path=/admin/collection&action=SPLITSHARD&collection=testCollection&shard=shard1&splitMethod=${method} [junit4] 2> 592336 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario RunSolrRequest path=/admin/collection&action=SPLITSHARD&collection=testCollection&shard=shard1&splitMethod=REWRITE [junit4] 2> 592339 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting. [junit4] 2> 592340 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now [junit4] 2> 592340 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now [junit4] 2> 592340 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor [junit4] 2> 592340 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor [junit4] 2> 592340 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely [junit4] 2> 592340 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly [junit4] 2> 592340 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 1. CreateCluster numNodes=2 [junit4] 2> 592340 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario CreateCluster numNodes=2 [junit4] 2> 592349 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10008_solr [junit4] 2> 592349 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory. [junit4] 2> 592349 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 2. LoadAutoscaling json={'cluster-policy':[]} [junit4] 2> 592349 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario LoadAutoscaling json={'cluster-policy':[]} [junit4] 2> 592350 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers [junit4] 2> 592350 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1 [junit4] 2> 592351 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0 [junit4] 2> 592351 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1 [junit4] 2> 592351 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10009_solr, 127.0.0.1:10008_solr] [junit4] 2> 592351 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded markers [junit4] 2> 592351 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1 [junit4] 2> 592351 DEBUG (ScheduledTrigger-5138-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592355 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 3. RunSolrRequest path=/admin/collections&action=CREATE&name=testCollection&numShards=2&replicationFactor=2&maxShardsPerNode=5 [junit4] 2> 592355 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario RunSolrRequest path=/admin/collections&action=CREATE&name=testCollection&numShards=2&replicationFactor=2&maxShardsPerNode=5 [junit4] 2> 592356 DEBUG (simCloudManagerPool-5137-thread-1) [ ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection testCollection [junit4] 2> 592367 DEBUG (simCloudManagerPool-5137-thread-1) [ ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection testCollection [junit4] 2> 592367 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 4. WaitCollection collection=testCollection&shards=2&replicas=2 [junit4] 2> 592367 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario WaitCollection collection=testCollection&shards=2&replicas=2 [junit4] 2> 592369 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.CloudUtil -- shards without leaders: [shard2, shard1] [junit4] 2> 592371 DEBUG (ScheduledTrigger-5138-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592379 DEBUG (simCloudManagerPool-5137-thread-5) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard2) [junit4] 2> 592380 DEBUG (simCloudManagerPool-5137-thread-5) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard2 (currentVersion=3): {"core_node4":{ [junit4] 2> "core":"testCollection_shard2_replica_n4", [junit4] 2> "shard":"shard2", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10008_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":10240, [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":9.5367431640625E-6, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 592391 DEBUG (simCloudManagerPool-5137-thread-2) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard1) [junit4] 2> 592392 DEBUG (simCloudManagerPool-5137-thread-2) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard1 (currentVersion=4): {"core_node1":{ [junit4] 2> "core":"testCollection_shard1_replica_n1", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10009_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":10240, [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":9.5367431640625E-6, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 592392 DEBUG (ScheduledTrigger-5138-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592413 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 5. SetShardMetrics collection=testCollection&shard=shard1&INDEX.sizeInBytes=1000000000 [junit4] 2> 592413 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario SetShardMetrics collection=testCollection&shard=shard1&INDEX.sizeInBytes=1000000000 [junit4] 2> 592413 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 6. SetNodeMetrics nodeset=#ANY&freedisk=1.5 [junit4] 2> 592413 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario SetNodeMetrics nodeset=#ANY&freedisk=1.5 [junit4] 2> 592413 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 7. RunSolrRequest path=/admin/collection&action=SPLITSHARD&collection=testCollection&shard=shard1&splitMethod=${method} [junit4] 2> 592413 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario RunSolrRequest path=/admin/collection&action=SPLITSHARD&collection=testCollection&shard=shard1&splitMethod=LINK [junit4] 2> 592413 DEBUG (ScheduledTrigger-5138-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592434 DEBUG (ScheduledTrigger-5138-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592455 DEBUG (ScheduledTrigger-5138-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592476 DEBUG (ScheduledTrigger-5138-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592497 DEBUG (ScheduledTrigger-5138-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592514 INFO (simCloudManagerPool-5137-thread-6) [ ] o.a.s.c.a.c.SplitShardCmd numSubShards set at: 2 [junit4] 2> 592518 DEBUG (ScheduledTrigger-5138-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592539 DEBUG (ScheduledTrigger-5138-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592559 DEBUG (simCloudManagerPool-5137-thread-9) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard1_1) [junit4] 2> 592561 DEBUG (ScheduledTrigger-5138-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 592560 DEBUG (simCloudManagerPool-5137-thread-9) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard1_1 (currentVersion=8): {"core_node12":{ [junit4] 2> "core":"testCollection_shard1_1_replica_n11", [junit4] 2> "shard":"shard1_1", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10009_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":10240, [junit4] 2> "base_url":"http://127.0.0.1:10009/solr", [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":9.5367431640625E-6, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 592562 DEBUG (simCloudManagerPool-5137-thread-10) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard1_0) [junit4] 2> 592562 DEBUG (simCloudManagerPool-5137-thread-10) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard1_0 (currentVersion=9): {"core_node10":{ [junit4] 2> "core":"testCollection_shard1_0_replica_n9", [junit4] 2> "shard":"shard1_0", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10008_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":20480, [junit4] 2> "base_url":"http://127.0.0.1:10008/solr", [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":1.9073486328125E-5, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 592562 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 8. WaitCollection collection=testCollection&shards=4&withInactive=true&replicas=2&requireLeaders=true [junit4] 2> 592562 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario WaitCollection collection=testCollection&shards=4&withInactive=true&replicas=2&requireLeaders=true [junit4] 2> 592565 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting. [junit4] 2> 592571 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now [junit4] 2> 592572 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now [junit4] 2> 592572 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor [junit4] 2> 592572 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor [junit4] 2> 592572 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely [junit4] 2> 592572 DEBUG (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly [junit4] 2> 592572 INFO (TEST-TestSimScenario.testSplitShard-seed#[EA19F5D72A768827]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testSplitShard [junit4] 2> 592574 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testIndexing [junit4] 2> 592574 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 1. CreateCluster numNodes=100 [junit4] 2> 592574 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario CreateCluster numNodes=100 [junit4] 2> 592585 DEBUG (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10010_solr [junit4] 2> 592585 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory. [junit4] 2> 592586 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 2. LoadAutoscaling json={'cluster-policy':[]} [junit4] 2> 592586 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario LoadAutoscaling json={'cluster-policy':[]} [junit4] 2> 592586 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1 [junit4] 2> 592586 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0 [junit4] 2> 592586 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1 [junit4] 2> 592586 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10082_solr, 127.0.0.1:10018_solr, 127.0.0.1:10049_solr, 127.0.0.1:10104_solr, 127.0.0.1:10020_solr, 127.0.0.1:10065_solr, 127.0.0.1:10021_solr, 127.0.0.1:10066_solr, 127.0.0.1:10035_solr, 127.0.0.1:10051_solr, 127.0.0.1:10019_solr, 127.0.0.1:10096_solr, 127.0.0.1:10022_solr, 127.0.0.1:10033_solr, 127.0.0.1:10105_solr, 127.0.0.1:10017_solr, 127.0.0.1:10047_solr, 127.0.0.1:10102_solr, 127.0.0.1:10095_solr, 127.0.0.1:10081_solr, 127.0.0.1:10084_solr, 127.0.0.1:10036_solr, 127.0.0.1:10052_solr, 127.0.0.1:10079_solr, 127.0.0.1:10098_solr, 127.0.0.1:10054_solr, 127.0.0.1:10099_solr, 127.0.0.1:10101_solr, 127.0.0.1:10071_solr, 127.0.0.1:10023_solr, 127.0.0.1:10076_solr, 127.0.0.1:10107_solr, 127.0.0.1:10055_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10070_solr, 127.0.0.1:10077_solr, 127.0.0.1:10038_solr, 127.0.0.1:10032_solr, 127.0.0.1:10108_solr, 127.0.0.1:10093_solr, 127.0.0.1:10061_solr, 127.0.0.1:10067_solr, 127.0.0.1:10086_solr, 127.0.0.1:10089_solr, 127.0.0.1:10045_solr, 127.0.0.1:10042_solr, 127.0.0.1:10029_solr, 127.0.0.1:10083_solr, 127.0.0.1:10080_solr, 127.0.0.1:10048_solr, 127.0.0.1:10064_solr, 127.0.0.1:10043_solr, 127.0.0.1:10057_solr, 127.0.0.1:10074_solr, 127.0.0.1:10026_solr, 127.0.0.1:10012_solr, 127.0.0.1:10060_solr, 127.0.0.1:10073_solr, 127.0.0.1:10058_solr, 127.0.0.1:10013_solr, 127.0.0.1:10090_solr, 127.0.0.1:10088_solr, 127.0.0.1:10027_solr, 127.0.0.1:10039_solr, 127.0.0.1:10025_solr, 127.0.0.1:10044_solr, 127.0.0.1:10028_solr, 127.0.0.1:10092_solr, 127.0.0.1:10041_solr, 127.0.0.1:10087_solr, 127.0.0.1:10030_solr, 127.0.0.1:10014_solr, 127.0.0.1:10011_solr, 127.0.0.1:10040_solr, 127.0.0.1:10085_solr, 127.0.0.1:10046_solr, 127.0.0.1:10068_solr, 127.0.0.1:10063_solr, 127.0.0.1:10015_solr, 127.0.0.1:10069_solr, 127.0.0.1:10062_solr, 127.0.0.1:10024_solr, 127.0.0.1:10050_solr, 127.0.0.1:10075_solr, 127.0.0.1:10078_solr, 127.0.0.1:10097_solr, 127.0.0.1:10031_solr, 127.0.0.1:10034_solr, 127.0.0.1:10091_solr, 127.0.0.1:10094_solr, 127.0.0.1:10106_solr, 127.0.0.1:10100_solr, 127.0.0.1:10037_solr, 127.0.0.1:10109_solr, 127.0.0.1:10053_solr, 127.0.0.1:10059_solr, 127.0.0.1:10103_solr, 127.0.0.1:10056_solr, 127.0.0.1:10072_solr] [junit4] 2> 592586 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded markers [junit4] 2> 592586 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1 [junit4] 2> 592586 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592591 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 3. RunSolrRequest path=/admin/collections&action=CREATE&autoAddReplicas=true&name=testCollection&numShards=2&replicationFactor=2&maxShardsPerNode=2 [junit4] 2> 592591 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario RunSolrRequest path=/admin/collections&action=CREATE&autoAddReplicas=true&name=testCollection&numShards=2&replicationFactor=2&maxShardsPerNode=2 [junit4] 2> 592591 DEBUG (simCloudManagerPool-5143-thread-1) [ ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection testCollection [junit4] 2> 592602 DEBUG (simCloudManagerPool-5143-thread-1) [ ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection testCollection [junit4] 2> 592619 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 4. WaitCollection collection=testCollection&shards=2&replicas=2 [junit4] 2> 592619 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario WaitCollection collection=testCollection&shards=2&replicas=2 [junit4] 2> 592622 DEBUG (ScheduledTrigger-5144-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592636 DEBUG (simCloudManagerPool-5143-thread-2) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard1) [junit4] 2> 592636 DEBUG (simCloudManagerPool-5143-thread-2) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard1 (currentVersion=2): {"core_node1":{ [junit4] 2> "core":"testCollection_shard1_replica_n1", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10099_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":10240, [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":9.5367431640625E-6, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 592646 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592653 DEBUG (simCloudManagerPool-5143-thread-5) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard2) [junit4] 2> 592653 DEBUG (simCloudManagerPool-5143-thread-5) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard2 (currentVersion=3): {"core_node4":{ [junit4] 2> "core":"testCollection_shard2_replica_n4", [junit4] 2> "shard":"shard2", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10059_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":10240, [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":9.5367431640625E-6, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 592667 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592679 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 5. RunSolrRequest path=/admin/autoscaling&httpMethod=POST&stream.body={'set-trigger':{'name':'indexSizeTrigger','event':'indexSize','waitFor':'10s','aboveDocs':1000,'enabled':true,'actions':[{'name':'compute_plan','class':'solr.ComputePlanAction'},{'name':'execute_plan','class':'solr.ExecutePlanAction'}]}} [junit4] 2> 592679 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario RunSolrRequest path=/admin/autoscaling&httpMethod=POST&stream.body={'set-trigger':{'name':'indexSizeTrigger','event':'indexSize','waitFor':'10s','aboveDocs':1000,'enabled':true,'actions':[{'name':'compute_plan','class':'solr.ComputePlanAction'},{'name':'execute_plan','class':'solr.ExecutePlanAction'}]}} [junit4] 2> 592691 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592718 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592749 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592775 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592779 DEBUG (simCloudManagerPool-5143-thread-6) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration [junit4] 2> 592780 DEBUG (simCloudManagerPool-5143-thread-6) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2 [junit4] 2> 592780 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2 [junit4] 2> 592780 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded markers [junit4] 2> 592780 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2 [junit4] 2> 592780 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 6. SetEventListener trigger=indexSizeTrigger&stage=SUCCEEDED [junit4] 2> 592780 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario SetEventListener trigger=indexSizeTrigger&stage=SUCCEEDED [junit4] 2> 592782 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 7. IndexDocs collection=testCollection&numDocs=3000 [junit4] 2> 592782 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario IndexDocs collection=testCollection&numDocs=3000 [junit4] 2> 592799 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592799 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 8. RunSimulator [junit4] 2> 592799 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario RunSimulator [junit4] 2> 592820 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592841 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592865 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592885 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592906 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592928 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592948 DEBUG (ScheduledTrigger-5144-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592969 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 592989 DEBUG (ScheduledTrigger-5144-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593010 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593030 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593035 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: { [junit4] 2> "id":"152c3b80969f4T2ou2v0rnxsao10xz7pvnk072j", [junit4] 2> "source":"indexSizeTrigger", [junit4] 2> "eventTime":372475536435700, [junit4] 2> "eventType":"INDEXSIZE", [junit4] 2> "properties":{ [junit4] 2> "aboveSize":{ [junit4] 2> "testCollection_shard1_replica_n1":"docs=1500, bytes=3082240", [junit4] 2> "testCollection_shard2_replica_n4":"docs=1500, bytes=3082240"}, [junit4] 2> "belowSize":{}, [junit4] 2> "requestedOps":[ [junit4] 2> { [junit4] 2> "action":"SPLITSHARD", [junit4] 2> "hints":{ [junit4] 2> "COLL_SHARD":[{ [junit4] 2> "first":"testCollection", [junit4] 2> "second":"shard1"}], [junit4] 2> "PARAMS":{ [junit4] 2> "splitByPrefix":false, [junit4] 2> "splitMethod":"link"}}}, [junit4] 2> { [junit4] 2> "action":"SPLITSHARD", [junit4] 2> "hints":{ [junit4] 2> "COLL_SHARD":[{ [junit4] 2> "first":"testCollection", [junit4] 2> "second":"shard2"}], [junit4] 2> "PARAMS":{ [junit4] 2> "splitByPrefix":false, [junit4] 2> "splitMethod":"link"}}}]}} [junit4] 2> 593037 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, .scheduled_maintenance, indexSizeTrigger] [junit4] 2> 593037 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event { [junit4] 2> "id":"152c3b80969f4T2ou2v0rnxsao10xz7pvnk072j", [junit4] 2> "source":"indexSizeTrigger", [junit4] 2> "eventTime":372475536435700, [junit4] 2> "eventType":"INDEXSIZE", [junit4] 2> "properties":{ [junit4] 2> "aboveSize":{ [junit4] 2> "testCollection_shard1_replica_n1":"docs=1500, bytes=3082240", [junit4] 2> "testCollection_shard2_replica_n4":"docs=1500, bytes=3082240"}, [junit4] 2> "belowSize":{}, [junit4] 2> "_enqueue_time_":372486342425700, [junit4] 2> "requestedOps":[ [junit4] 2> { [junit4] 2> "action":"SPLITSHARD", [junit4] 2> "hints":{ [junit4] 2> "COLL_SHARD":[{ [junit4] 2> "first":"testCollection", [junit4] 2> "second":"shard1"}], [junit4] 2> "PARAMS":{ [junit4] 2> "splitByPrefix":false, [junit4] 2> "splitMethod":"link"}}}, [junit4] 2> { [junit4] 2> "action":"SPLITSHARD", [junit4] 2> "hints":{ [junit4] 2> "COLL_SHARD":[{ [junit4] 2> "first":"testCollection", [junit4] 2> "second":"shard2"}], [junit4] 2> "PARAMS":{ [junit4] 2> "splitByPrefix":false, [junit4] 2> "splitMethod":"link"}}}]}} [junit4] 2> 593037 DEBUG (AutoscalingActionExecutor-5145-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for { [junit4] 2> "id":"152c3b80969f4T2ou2v0rnxsao10xz7pvnk072j", [junit4] 2> "source":"indexSizeTrigger", [junit4] 2> "eventTime":372475536435700, [junit4] 2> "eventType":"INDEXSIZE", [junit4] 2> "properties":{ [junit4] 2> "aboveSize":{ [junit4] 2> "testCollection_shard1_replica_n1":"docs=1500, bytes=3082240", [junit4] 2> "testCollection_shard2_replica_n4":"docs=1500, bytes=3082240"}, [junit4] 2> "belowSize":{}, [junit4] 2> "_enqueue_time_":372486342425700, [junit4] 2> "requestedOps":[ [junit4] 2> { [junit4] 2> "action":"SPLITSHARD", [junit4] 2> "hints":{ [junit4] 2> "COLL_SHARD":[{ [junit4] 2> "first":"testCollection", [junit4] 2> "second":"shard1"}], [junit4] 2> "PARAMS":{ [junit4] 2> "splitByPrefix":false, [junit4] 2> "splitMethod":"link"}}}, [junit4] 2> { [junit4] 2> "action":"SPLITSHARD", [junit4] 2> "hints":{ [junit4] 2> "COLL_SHARD":[{ [junit4] 2> "first":"testCollection", [junit4] 2> "second":"shard2"}], [junit4] 2> "PARAMS":{ [junit4] 2> "splitByPrefix":false, [junit4] 2> "splitMethod":"link"}}}]}} [junit4] 2> 593037 DEBUG (AutoscalingActionExecutor-5145-thread-1) [ ] o.a.s.c.a.ComputePlanAction -- processing event: { [junit4] 2> "id":"152c3b80969f4T2ou2v0rnxsao10xz7pvnk072j", [junit4] 2> "source":"indexSizeTrigger", [junit4] 2> "eventTime":372475536435700, [junit4] 2> "eventType":"INDEXSIZE", [junit4] 2> "properties":{ [junit4] 2> "aboveSize":{ [junit4] 2> "testCollection_shard1_replica_n1":"docs=1500, bytes=3082240", [junit4] 2> "testCollection_shard2_replica_n4":"docs=1500, bytes=3082240"}, [junit4] 2> "belowSize":{}, [junit4] 2> "_enqueue_time_":372486342425700, [junit4] 2> "requestedOps":[ [junit4] 2> { [junit4] 2> "action":"SPLITSHARD", [junit4] 2> "hints":{ [junit4] 2> "COLL_SHARD":[{ [junit4] 2> "first":"testCollection", [junit4] 2> "second":"shard1"}], [junit4] 2> "PARAMS":{ [junit4] 2> "splitByPrefix":false, [junit4] 2> "splitMethod":"link"}}}, [junit4] 2> { [junit4] 2> "action":"SPLITSHARD", [junit4] 2> "hints":{ [junit4] 2> "COLL_SHARD":[{ [junit4] 2> "first":"testCollection", [junit4] 2> "second":"shard2"}], [junit4] 2> "PARAMS":{ [junit4] 2> "splitByPrefix":false, [junit4] 2> "splitMethod":"link"}}}]}} with context properties: {BEFORE_ACTION=[compute_plan]} [junit4] 2> 593039 DEBUG (AutoscalingActionExecutor-5145-thread-1) [ ] o.a.s.c.a.ComputePlanAction Computed Plan: action=SPLITSHARD&collection=testCollection&shard=shard1&splitMethod=link&splitByPrefix=false [junit4] 2> 593039 DEBUG (AutoscalingActionExecutor-5145-thread-1) [ ] o.a.s.c.a.ComputePlanAction Computed Plan: action=SPLITSHARD&collection=testCollection&shard=shard2&splitMethod=link&splitByPrefix=false [junit4] 2> 593039 DEBUG (AutoscalingActionExecutor-5145-thread-1) [ ] o.a.s.c.a.ExecutePlanAction -- processing event: { [junit4] 2> "id":"152c3b80969f4T2ou2v0rnxsao10xz7pvnk072j", [junit4] 2> "source":"indexSizeTrigger", [junit4] 2> "eventTime":372475536435700, [junit4] 2> "eventType":"INDEXSIZE", [junit4] 2> "properties":{ [junit4] 2> "__start__":2, [junit4] 2> "aboveSize":{ [junit4] 2> "testCollection_shard1_replica_n1":"docs=1500, bytes=3082240", [junit4] 2> "testCollection_shard2_replica_n4":"docs=1500, bytes=3082240"}, [junit4] 2> "belowSize":{}, [junit4] 2> "_enqueue_time_":372486342425700, [junit4] 2> "requestedOps":[ [junit4] 2> { [junit4] 2> "action":"SPLITSHARD", [junit4] 2> "hints":{ [junit4] 2> "COLL_SHARD":[{ [junit4] 2> "first":"testCollection", [junit4] 2> "second":"shard1"}], [junit4] 2> "PARAMS":{ [junit4] 2> "splitByPrefix":false, [junit4] 2> "splitMethod":"link"}}}, [junit4] 2> { [junit4] 2> "action":"SPLITSHARD", [junit4] 2> "hints":{ [junit4] 2> "COLL_SHARD":[{ [junit4] 2> "first":"testCollection", [junit4] 2> "second":"shard2"}], [junit4] 2> "PARAMS":{ [junit4] 2> "splitByPrefix":false, [junit4] 2> "splitMethod":"link"}}}]}} with context properties: {operations=[{ [junit4] 2> "class":"org.apache.solr.client.solrj.request.CollectionAdminRequest$SplitShard", [junit4] 2> "method":"GET", [junit4] 2> "params.action":"SPLITSHARD", [junit4] 2> "params.collection":"testCollection", [junit4] 2> "params.shard":"shard1", [junit4] 2> "params.splitMethod":"link", [junit4] 2> "params.splitByPrefix":"false"}, { [junit4] 2> "class":"org.apache.solr.client.solrj.request.CollectionAdminRequest$SplitShard", [junit4] 2> "method":"GET", [junit4] 2> "params.action":"SPLITSHARD", [junit4] 2> "params.collection":"testCollection", [junit4] 2> "params.shard":"shard2", [junit4] 2> "params.splitMethod":"link", [junit4] 2> "params.splitByPrefix":"false"}], AFTER_ACTION=[compute_plan], BEFORE_ACTION=[compute_plan, execute_plan]} [junit4] 2> 593039 DEBUG (AutoscalingActionExecutor-5145-thread-1) [ ] o.a.s.c.a.ExecutePlanAction Executing operation: action=SPLITSHARD&collection=testCollection&shard=shard1&splitMethod=link&splitByPrefix=false [junit4] 2> 593139 INFO (simCloudManagerPool-5143-thread-8) [ ] o.a.s.c.a.c.SplitShardCmd numSubShards set at: 2 [junit4] 2> 593199 DEBUG (simCloudManagerPool-5143-thread-8) [ ] o.a.s.c.a.s.SimClusterStateProvider ** no leader in testCollection / shard1_0:{ [junit4] 2> "parent":"shard1", [junit4] 2> "stateTimestamp":"1590089123184055000", [junit4] 2> "range":"80000000-bfffffff", [junit4] 2> "state":"construction", [junit4] 2> "replicas":{ [junit4] 2> "core_node10":{ [junit4] 2> "core":"testCollection_shard1_0_replica_n9", [junit4] 2> "SEARCHER.searcher.maxDoc":750, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":1556480, [junit4] 2> "node_name":"127.0.0.1:10018_solr", [junit4] 2> "base_url":"http://127.0.0.1:10018/solr", [junit4] 2> "state":"active", [junit4] 2> "type":"NRT", [junit4] 2> "INDEX.sizeInGB":0.0014495849609375, [junit4] 2> "SEARCHER.searcher.numDocs":750}, [junit4] 2> "core_node8":{ [junit4] 2> "core":"testCollection_shard1_0_replica_n7", [junit4] 2> "SEARCHER.searcher.maxDoc":750, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":1556480, [junit4] 2> "node_name":"127.0.0.1:10082_solr", [junit4] 2> "base_url":"http://127.0.0.1:10082/solr", [junit4] 2> "state":"active", [junit4] 2> "type":"NRT", [junit4] 2> "INDEX.sizeInGB":0.0014495849609375, [junit4] 2> "SEARCHER.searcher.numDocs":750}}} [junit4] 2> 593199 DEBUG (simCloudManagerPool-5143-thread-12) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard1_0) [junit4] 2> 593204 DEBUG (simCloudManagerPool-5143-thread-12) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard1_0 (currentVersion=8): {"core_node10":{ [junit4] 2> "core":"testCollection_shard1_0_replica_n9", [junit4] 2> "shard":"shard1_0", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10018_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":750, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":1556480, [junit4] 2> "base_url":"http://127.0.0.1:10018/solr", [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":0.0014495849609375, [junit4] 2> "SEARCHER.searcher.numDocs":750}} [junit4] 2> 593206 DEBUG (simCloudManagerPool-5143-thread-11) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard1_1) [junit4] 2> 593206 DEBUG (simCloudManagerPool-5143-thread-11) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard1_1 (currentVersion=10): {"core_node12":{ [junit4] 2> "core":"testCollection_shard1_1_replica_n11", [junit4] 2> "shard":"shard1_1", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10049_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":750, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":1546240, [junit4] 2> "base_url":"http://127.0.0.1:10049/solr", [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":0.0014400482177734375, [junit4] 2> "SEARCHER.searcher.numDocs":750}} [junit4] 2> 593210 DEBUG (AutoscalingActionExecutor-5145-thread-1) [ ] o.a.s.c.a.ExecutePlanAction Executing operation: action=SPLITSHARD&collection=testCollection&shard=shard2&splitMethod=link&splitByPrefix=false [junit4] 2> 593319 INFO (simCloudManagerPool-5143-thread-15) [ ] o.a.s.c.a.c.SplitShardCmd numSubShards set at: 2 [junit4] 2> 593380 DEBUG (simCloudManagerPool-5143-thread-18) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard2_1) [junit4] 2> 593380 DEBUG (simCloudManagerPool-5143-thread-18) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard2_1 (currentVersion=16): {"core_node22":{ [junit4] 2> "core":"testCollection_shard2_1_replica_n21", [junit4] 2> "shard":"shard2_1", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10049_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":750, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":1546240, [junit4] 2> "base_url":"http://127.0.0.1:10049/solr", [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":0.0014400482177734375, [junit4] 2> "SEARCHER.searcher.numDocs":750}} [junit4] 2> 593381 DEBUG (simCloudManagerPool-5143-thread-15) [ ] o.a.s.c.a.s.SimClusterStateProvider ** no leader in testCollection / shard2_0:{ [junit4] 2> "parent":"shard2", [junit4] 2> "stateTimestamp":"1590089132689255000", [junit4] 2> "range":"0-3fffffff", [junit4] 2> "state":"construction", [junit4] 2> "replicas":{ [junit4] 2> "core_node18":{ [junit4] 2> "core":"testCollection_shard2_0_replica_n17", [junit4] 2> "SEARCHER.searcher.maxDoc":750, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":1556480, [junit4] 2> "node_name":"127.0.0.1:10082_solr", [junit4] 2> "base_url":"http://127.0.0.1:10082/solr", [junit4] 2> "state":"active", [junit4] 2> "type":"NRT", [junit4] 2> "INDEX.sizeInGB":0.0014495849609375, [junit4] 2> "SEARCHER.searcher.numDocs":750}, [junit4] 2> "core_node20":{ [junit4] 2> "core":"testCollection_shard2_0_replica_n19", [junit4] 2> "SEARCHER.searcher.maxDoc":750, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":1556480, [junit4] 2> "node_name":"127.0.0.1:10018_solr", [junit4] 2> "base_url":"http://127.0.0.1:10018/solr", [junit4] 2> "state":"active", [junit4] 2> "type":"NRT", [junit4] 2> "INDEX.sizeInGB":0.0014495849609375, [junit4] 2> "SEARCHER.searcher.numDocs":750}}} [junit4] 2> 593381 DEBUG (simCloudManagerPool-5143-thread-21) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard2_0) [junit4] 2> 593381 DEBUG (simCloudManagerPool-5143-thread-21) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard2_0 (currentVersion=17): {"core_node18":{ [junit4] 2> "core":"testCollection_shard2_0_replica_n17", [junit4] 2> "shard":"shard2_0", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10082_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":750, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":1556480, [junit4] 2> "base_url":"http://127.0.0.1:10082/solr", [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":0.0014495849609375, [junit4] 2> "SEARCHER.searcher.numDocs":750}} [junit4] 2> 593404 DEBUG (AutoscalingActionExecutor-5145-thread-1) [ ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event { [junit4] 2> "id":"152c3b80969f4T2ou2v0rnxsao10xz7pvnk072j", [junit4] 2> "source":"indexSizeTrigger", [junit4] 2> "eventTime":372475536435700, [junit4] 2> "eventType":"INDEXSIZE", [junit4] 2> "properties":{ [junit4] 2> "__start__":2, [junit4] 2> "aboveSize":{ [junit4] 2> "testCollection_shard1_replica_n1":"docs=1500, bytes=3082240", [junit4] 2> "testCollection_shard2_replica_n4":"docs=1500, bytes=3082240"}, [junit4] 2> "belowSize":{}, [junit4] 2> "_enqueue_time_":372486342425700, [junit4] 2> "requestedOps":[ [junit4] 2> { [junit4] 2> "action":"SPLITSHARD", [junit4] 2> "hints":{ [junit4] 2> "COLL_SHARD":[{ [junit4] 2> "first":"testCollection", [junit4] 2> "second":"shard1"}], [junit4] 2> "PARAMS":{ [junit4] 2> "splitByPrefix":false, [junit4] 2> "splitMethod":"link"}}}, [junit4] 2> { [junit4] 2> "action":"SPLITSHARD", [junit4] 2> "hints":{ [junit4] 2> "COLL_SHARD":[{ [junit4] 2> "first":"testCollection", [junit4] 2> "second":"shard2"}], [junit4] 2> "PARAMS":{ [junit4] 2> "splitByPrefix":false, [junit4] 2> "splitMethod":"link"}}}]}} [junit4] 2> 593404 DEBUG (AutoscalingActionExecutor-5145-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .scheduled_maintenance after 100ms [junit4] 2> 593404 DEBUG (AutoscalingActionExecutor-5145-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms [junit4] 2> 593404 DEBUG (AutoscalingActionExecutor-5145-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: indexSizeTrigger after 100ms [junit4] 2> 593404 DEBUG (AutoscalingActionExecutor-5145-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 18353 ms for event id=152c3b80969f4T2ou2v0rnxsao10xz7pvnk072j [junit4] 2> 593510 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593537 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593561 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593582 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593603 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593626 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593648 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593669 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593690 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593711 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593738 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593761 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593784 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593806 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593826 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593847 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593869 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593890 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593911 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593931 DEBUG (ScheduledTrigger-5144-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593952 DEBUG (ScheduledTrigger-5144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593974 DEBUG (ScheduledTrigger-5144-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593994 DEBUG (ScheduledTrigger-5144-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 100 and last live nodes: 100 [junit4] 2> 593999 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 9. WaitEvent trigger=indexSizeTrigger&wait=60 [junit4] 2> 593999 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario WaitEvent trigger=indexSizeTrigger&wait=60 [junit4] 2> 594000 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 10. Assert condition=not_null&key=_trigger_event_indexSizeTrigger [junit4] 2> 594000 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario Assert condition=not_null&key=_trigger_event_indexSizeTrigger [junit4] 2> 594001 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 11. Assert condition=equals&key=_trigger_event_indexSizeTrigger/eventType&expected=INDEXSIZE [junit4] 2> 594001 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario Assert condition=equals&key=_trigger_event_indexSizeTrigger/eventType&expected=INDEXSIZE [junit4] 2> 594001 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 12. Assert condition=equals&key=_trigger_event_indexSizeTrigger/properties/requestedOps[0]/action&expected=SPLITSHARD [junit4] 2> 594001 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario Assert condition=equals&key=_trigger_event_indexSizeTrigger/properties/requestedOps[0]/action&expected=SPLITSHARD [junit4] 2> 594001 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 13. WaitCollection collection=testCollection&shards=6&withInactive=true&requireLeaders=false&replicas=2 [junit4] 2> 594001 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario WaitCollection collection=testCollection&shards=6&withInactive=true&requireLeaders=false&replicas=2 [junit4] 2> 594002 DEBUG (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now [junit4] 2> 594002 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting. [junit4] 2> 594002 DEBUG (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now [junit4] 2> 594002 DEBUG (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor [junit4] 2> 594002 DEBUG (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor [junit4] 2> 594002 DEBUG (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely [junit4] 2> 594002 DEBUG (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly [junit4] 2> 594003 INFO (TEST-TestSimScenario.testIndexing-seed#[EA19F5D72A768827]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testIndexing [junit4] 2> 594005 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testAutoAddReplicas [junit4] 2> 594006 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 1. CreateCluster numNodes=2 [junit4] 2> 594006 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario CreateCluster numNodes=2 [junit4] 2> 594012 DEBUG (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10110_solr [junit4] 2> 594012 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory. [junit4] 2> 594012 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers [junit4] 2> 594012 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 2. LoadAutoscaling json={'cluster-policy'+:+[{'replica'+:+'<3',+'shard'+:+'#EACH',+'collection'+:+'testCollection','node':'#ANY'}]}&defaultWaitFor=10 [junit4] 2> 594012 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario LoadAutoscaling json={'cluster-policy'+:+[{'replica'+:+'<3',+'shard'+:+'#EACH',+'collection'+:+'testCollection','node':'#ANY'}]}&defaultWaitFor=10 [junit4] 2> 594012 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1 [junit4] 2> 594013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0 [junit4] 2> 594013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1 [junit4] 2> 594013 DEBUG (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2 [junit4] 2> 594013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10110_solr, 127.0.0.1:10111_solr] [junit4] 2> 594013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded markers [junit4] 2> 594013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 1 [junit4] 2> 594013 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2 [junit4] 2> 594014 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10110_solr, 127.0.0.1:10111_solr] [junit4] 2> 594014 DEBUG (ScheduledTrigger-5150-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 594014 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded markers [junit4] 2> 594014 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2 [junit4] 2> 594018 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 3. RunSolrRequest path=/admin/collections&action=CREATE&autoAddReplicas=true&name=testCollection&numShards=2&replicationFactor=2&maxShardsPerNode=2 [junit4] 2> 594018 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario RunSolrRequest path=/admin/collections&action=CREATE&autoAddReplicas=true&name=testCollection&numShards=2&replicationFactor=2&maxShardsPerNode=2 [junit4] 2> 594018 DEBUG (simCloudManagerPool-5149-thread-1) [ ] o.a.s.c.a.s.SimClusterStateProvider -- simCreateCollection testCollection [junit4] 2> 594034 DEBUG (ScheduledTrigger-5150-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 594039 DEBUG (simCloudManagerPool-5149-thread-2) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard1) [junit4] 2> 594039 DEBUG (simCloudManagerPool-5149-thread-2) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard1 (currentVersion=2): {"core_node1":{ [junit4] 2> "core":"testCollection_shard1_replica_n1", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10111_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":10240, [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":9.5367431640625E-6, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 594039 DEBUG (simCloudManagerPool-5149-thread-1) [ ] o.a.s.c.a.s.SimClusterStateProvider -- finished createCollection testCollection [junit4] 2> 594039 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 4. WaitCollection collection=testCollection&shards=2&replicas=2 [junit4] 2> 594039 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario WaitCollection collection=testCollection&shards=2&replicas=2 [junit4] 2> 594056 DEBUG (ScheduledTrigger-5150-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 594060 DEBUG (simCloudManagerPool-5149-thread-4) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard2) [junit4] 2> 594060 DEBUG (simCloudManagerPool-5149-thread-4) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard2 (currentVersion=4): {"core_node3":{ [junit4] 2> "core":"testCollection_shard2_replica_n3", [junit4] 2> "shard":"shard2", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10110_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":10240, [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":9.5367431640625E-6, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 594070 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 5. SetEventListener trigger=.auto_add_replicas&stage=SUCCEEDED [junit4] 2> 594070 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario SetEventListener trigger=.auto_add_replicas&stage=SUCCEEDED [junit4] 2> 594070 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 6. KillNodes node=${_random_node_} [junit4] 2> 594070 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario KillNodes node=127.0.0.1:10110_solr [junit4] 2> 594070 DEBUG (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimClusterStateProvider -- created marker: /autoscaling/nodeLost/127.0.0.1:10110_solr [junit4] 2> 594070 DEBUG (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10111_solr [junit4] 2> 594070 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario 7. WaitEvent trigger=.auto_add_replicas&wait=60 [junit4] 2> 594070 INFO (TEST-TestSimScenario.testAutoAddReplicas-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.s.SimScenario WaitEvent trigger=.auto_add_replicas&wait=60 [junit4] 2> 594070 DEBUG (simCloudManagerPool-5149-thread-6) [ ] o.a.s.c.a.s.SimClusterStateProvider Running leader election (testCollection / shard2) [junit4] 2> 594070 INFO (simCloudManagerPool-5149-thread-6) [ ] o.a.s.c.ActionThrottle Throttling leader attempts - waiting for 4444ms [junit4] 2> 594077 DEBUG (ScheduledTrigger-5150-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 2 [junit4] 2> 594077 DEBUG (ScheduledTrigger-5150-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10110_solr [junit4] 2> 594098 DEBUG (ScheduledTrigger-5150-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1 [junit4] 2> 594119 DEBUG (ScheduledTrigger-5150-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1 [junit4] 2> 594141 DEBUG (ScheduledTrigger-5150-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1 [junit4] 2> 594163 DEBUG (ScheduledTrigger-5150-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1 [junit4] 2> 594163 DEBUG (simCloudManagerPool-5149-thread-6) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testCollection / shard2 (currentVersion=7): {"core_node4":{ [junit4] 2> "core":"testCollection_shard2_replica_n4", [junit4] 2> "shard":"shard2", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10111_solr", [junit4] 2> "type":"NRT", [junit4] 2> "leader":"true", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":10240, [junit4] 2> "state":"active", [junit4] 2> "INDEX.sizeInGB":9.5367431640625E-6, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 594184 DEBUG (ScheduledTrigger-5150-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1 [junit4] 2> 594205 DEBUG (ScheduledTrigger-5150-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1 [junit4] 2> 594225 DEBUG (ScheduledTrigger-5150-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1 [junit4] 2> 594246 DEBUG (ScheduledTrigger-5150-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1 [junit4] 2> 594267 DEBUG (ScheduledTrigger-5150-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1 [junit4] 2> 594287 DEBUG (ScheduledTrigger-5150-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1 [junit4] 2> 594287 DEBUG (ScheduledTrigger-5150-thread-4) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10110_solr] [junit4] 2> 594287 DEBUG (ScheduledTrigger-5150-thread-4) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: { [junit4] 2> "id":"152d258c329e4T2ou2v0rnxsao10xz7pvnk072l", [junit4] 2> "source":".auto_add_replicas", [junit4] 2> "eventTime":372538362505700, [junit4] 2> "eventType":"NODELOST", [junit4] 2> "properties":{ [junit4] 2> "eventTimes":[372538362505700], [junit4] 2> "preferredOperation":"movereplica", [junit4] 2> "nodeNames":["127.0.0.1:10110_solr"]}} [junit4] 2> 594288 DEBUG (ScheduledTrigger-5150-thread-4) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, .scheduled_maintenance] [junit4] 2> 594288 DEBUG (ScheduledTrigger-5150-thread-4) [ ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event { [junit4] 2> "id":"152d258c329e4T2ou2v0rnxsao10xz7pvnk072l", [junit4] 2> "source":".auto_add_replicas", [junit4] 2> "eventTime":372538362505700, [junit4] 2> "eventType":"NODELOST", [junit4] 2> "properties":{ [junit4] 2> "eventTimes":[372538362505700], [junit4] 2> "preferredOperation":"movereplica", [junit4] 2> "_enqueue_time_":372548880715700, [junit4] 2> "nodeNames":["127.0.0.1:10110_solr"]}} [junit4] 2> 594288 DEBUG (AutoscalingActionExecutor-5151-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for { [junit4] 2> "id":"152d258c329e4T2ou2v0rnxsao10xz7pvnk072l", [junit4] 2> "source":".auto_add_replicas", [junit4] 2> "eventTime":372538362505700, [junit4] 2> "eventType":"NODELOST", [junit4] 2> "properties":{ [junit4] 2> "eventTimes":[372538362505700], [junit4] 2> "preferredOperation":"movereplica", [junit4] 2> "_enqueue_time_":372548880715700, [junit4] 2> "nodeNames":["127.0.0.1:10110_solr"]}} [junit4] 2> 594288 DEBUG (AutoscalingActionExecutor-5151-thread-1) [ ] o.a.s.c.a.ComputePlanAction -- processing event: { [junit4] 2> "id":"152d258c329e4T2ou2v0rnxsao10xz7pvnk072l", [junit4] 2> "source":".auto_add_replicas", [junit4] 2> "eventTime":372538362505700, [junit4] 2> "eventType":"NODELOST", [junit4] 2> "properties":{ [junit4] 2> "eventTimes":[372538362505700], [junit4] 2> "preferredOperation":"movereplica", [junit4] 2> "_enqueue_time_":372548880715700, [junit4] 2> "nodeNames":["127.0.0.1:10110_solr"]}} with context properties: {BEFORE_ACTION=[auto_add_replicas_plan]} [junit4] 2> 594288 DEBUG (AutoscalingActionExecutor-5151-thread-1) [ ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testCollection&targetNode=127.0.0.1:10111_solr&inPlaceMove=true&replica=core_node2 [junit4] 2> 594288 DEBUG (AutoscalingActionExecutor-5151-thread-1) [ ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testCollection&targetNode=127.0.0.1:10111_solr&inPlaceMove=true&replica=core_node3 [junit4] 2> 594289 DEBUG (AutoscalingActionExecutor-5151-thread-1) [ ] o.a.s.c.a.ExecutePlanAction -- processing event: { [junit4] 2> "id":"152d258c329e4T2ou2v0rnxsao10xz7pvnk072l", [junit4] 2> "source":".auto_add_replicas", [junit4] 2> "eventTime":372538362505700, [junit4] 2> "eventType":"NODELOST", [junit4] 2> "properties":{ [junit4] 2> "eventTimes":[372538362505700], [junit4] 2> "preferredOperation":"movereplica", [junit4] 2> "_enqueue_time_":372548880715700, [junit4] 2> "nodeNames":["127.0.0.1:10110_solr"]}} with context properties: {operations=[{ [junit4] 2> "class":"org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica", [junit4] 2> "method":"GET", [junit4] 2> "params.action":"MOVEREPLICA", [junit4] 2> "params.collection":"testCollection", [junit4] 2> "params.targetNode":"127.0.0.1:10111_solr", [junit4] 2> "params.inPlaceMove":"true", [junit4] 2> "params.replica":"core_node2"}, { [junit4] 2> "class":"org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica", [junit4] 2> "method":"GET", [junit4] 2> "params.action":"MOVEREPLICA", [junit4] 2> "params.collection":"testCollection", [junit4] 2> "params.targetNode":"127.0.0.1:10111_solr", [junit4] 2> "params.inPlaceMove":"true", [junit4] 2> "params.replica":"core_node3"}], AFTER_ACTION=[auto_add_replicas_plan], BEFORE_ACTION=[auto_add_replicas_plan, execute_plan]} [junit4] 2> 594289 DEBUG (AutoscalingActionExecutor-5151-thread-1) [ ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testCollection&targetNode=127.0.0.1:10111_solr&inPlaceMove=true&replica=core_node2 [junit4] 2> 594390 DEBUG (simCloudManagerPool-5149-thread-8) [ ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node6":{ [junit4] 2> "core":"testCollection_shard1_replica_n5", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10111_solr", [junit4] 2> "type":"NRT", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":10240, [junit4] 2> "state":"down", [junit4] 2> "INDEX.sizeInGB":9.5367431640625E-6, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 594502 DEBUG (AutoscalingActionExecutor-5151-thread-1) [ ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testCollection&targetNode=127.0.0.1:10111_solr&inPlaceMove=true&replica=core_node3 [junit4] 2> 594603 DEBUG (simCloudManagerPool-5149-thread-11) [ ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node8":{ [junit4] 2> "core":"testCollection_shard2_replica_n7", [junit4] 2> "shard":"shard2", [junit4] 2> "collection":"testCollection", [junit4] 2> "node_name":"127.0.0.1:10111_solr", [junit4] 2> "type":"NRT", [junit4] 2> "SEARCHER.searcher.maxDoc":0, [junit4] 2> "SEARCHER.searcher.deletedDocs":0, [junit4] 2> "INDEX.sizeInBytes":10240, [junit4] 2> "state":"down", [junit4] 2> "INDEX.sizeInGB":9.5367431640625E-6, [junit4] 2> "SEARCHER.searcher.numDocs":0}} [junit4] 2> 594715 DEBUG (AutoscalingActionExecutor-5151-thread-1) [ ] o.a.s.c.a.SystemLogListener Collection .system missing, skip sending event { [junit4] 2> "id":"152d258c329e4T2ou2v0rnxsao10xz7pvnk072l", [junit4] 2> "source":".auto_add_replicas", [junit4] 2> "eventTime":372538362505700, [junit4] 2> "eventType":"NODELOST", [junit4] 2> "properties":{ [junit4] 2> "eventTimes":[372538362505700], [junit4] 2> "preferredOperation":"movereplica", [junit4] 2> "_enqueue_time_":372548880715700, [junit4] 2> "nodeNames":["127.0.0.1:10110_solr"]}} [junit4] 2> 594715 DEBUG (AutoscalingActionExecutor-5151-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms [junit4] 2> 594715 DEBUG (AutoscalingActionExecutor-5151-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .scheduled_maintenance after 100ms [junit4] 2> 594715 DEBUG (AutoscalingActionExecutor-5151-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 21338 ms for event id=1 [...truncated too long message...] gger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595157 DEBUG (ScheduledTrigger-5162-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595178 DEBUG (ScheduledTrigger-5162-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595199 DEBUG (ScheduledTrigger-5162-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595220 DEBUG (ScheduledTrigger-5162-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595241 DEBUG (ScheduledTrigger-5162-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595261 DEBUG (ScheduledTrigger-5162-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595282 DEBUG (ScheduledTrigger-5162-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595302 DEBUG (ScheduledTrigger-5162-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595324 DEBUG (ScheduledTrigger-5162-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595345 DEBUG (ScheduledTrigger-5162-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595365 DEBUG (ScheduledTrigger-5162-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595387 DEBUG (ScheduledTrigger-5162-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595407 DEBUG (ScheduledTrigger-5162-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595428 DEBUG (ScheduledTrigger-5162-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595449 DEBUG (ScheduledTrigger-5162-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595471 DEBUG (ScheduledTrigger-5162-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595492 DEBUG (ScheduledTrigger-5162-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595513 DEBUG (ScheduledTrigger-5162-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2 [junit4] 2> 595516 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting. [junit4] 2> 595516 DEBUG (TEST-TestSimScenario.testSuggestions-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now [junit4] 2> 595516 DEBUG (TEST-TestSimScenario.testSuggestions-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now [junit4] 2> 595516 DEBUG (TEST-TestSimScenario.testSuggestions-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor [junit4] 2> 595516 DEBUG (TEST-TestSimScenario.testSuggestions-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor [junit4] 2> 595516 DEBUG (TEST-TestSimScenario.testSuggestions-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely [junit4] 2> 595516 DEBUG (TEST-TestSimScenario.testSuggestions-seed#[EA19F5D72A768827]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly [junit4] 2> 595517 INFO (TEST-TestSimScenario.testSuggestions-seed#[EA19F5D72A768827]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testSuggestions [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSimScenario -Dtests.method=testSuggestions -Dtests.seed=EA19F5D72A768827 -Dtests.slow=true -Dtests.locale=fr-MR -Dtests.timezone=Antarctica/McMurdo -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] ERROR 0.74s J2 | TestSimScenario.testSuggestions <<< [junit4] > Throwable #1: java.util.concurrent.TimeoutException: OverseerTriggerThread never caught up to the latest znodeVersion [junit4] > at __randomizedtesting.SeedInfo.seed([EA19F5D72A768827:4CC5CC22E442DFD9]:0) [junit4] > at org.apache.solr.util.TimeOut.waitFor(TimeOut.java:66) [junit4] > at org.apache.solr.cloud.autoscaling.sim.SimScenario$LoadAutoscaling.execute(SimScenario.java:466) [junit4] > at org.apache.solr.cloud.autoscaling.sim.SimScenario.run(SimScenario.java:1110) [junit4] > at org.apache.solr.cloud.autoscaling.sim.TestSimScenario.testSuggestions(TestSimScenario.java:114) [junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:567) [junit4] > at java.base/java.lang.Thread.run(Thread.java:835) [junit4] 2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J2\temp\solr.cloud.autoscaling.sim.TestSimScenario_EA19F5D72A768827-001 [junit4] 2> NOTE: test params are: codec=Asserting(Lucene84), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@3e294ca6), locale=fr-MR, timezone=Antarctica/McMurdo [junit4] 2> NOTE: Windows 10 10.0 amd64/AdoptOpenJDK 12.0.2 (64-bit)/cpus=6,threads=1,free=18294024,total=205365248 [junit4] 2> NOTE: All tests run in this JVM: [HdfsCollectionsAPIDistributedZkTest, SolrSlf4jReporterTest, BigEndianAscendingWordSerializerTest, BadComponentTest, SolrPluginUtilsTest, RegexBoostProcessorTest, DistributedFacetSimpleRefinementLongTailTest, HdfsAutoAddReplicasIntegrationTest, ConcurrentCreateRoutedAliasTest, TestConfigSets, HttpPartitionOnCommitTest, TestUpdate, TestRebalanceLeaders, TestCSVResponseWriter, MetricTriggerIntegrationTest, DirectSolrSpellCheckerTest, ActionThrottleTest, OverseerModifyCollectionTest, LegacyCloudClusterPropTest, DistributedVersionInfoTest, TestStressCloudBlindAtomicUpdates, TestCollectionsAPIViaSolrCloudCluster, TestReloadDeadlock, RegexBytesRefFilterTest, TestManagedSchemaAPI, TestUnifiedSolrHighlighter, TestRTimerTree, UpdateRequestProcessorFactoryTest, NestedShardedAtomicUpdateTest, HLLUtilTest, TestCloudJSONFacetSKGEquiv, TestSizeLimitedDistributedMap, TestSolrJ, AutoAddReplicasIntegrationTest, TestHashPartitioner, TestHttpServletCarrier, TestSimComputePlanAction, TestSimpleTextCodec, TestIntervalFaceting, LeaderElectionTest, TestConfigSetsAPIZkFailure, RecoveryZkTest, IndexSchemaRuntimeFieldTest, TestInfoStreamLogging, TestSolrQueryResponse, AddSchemaFieldsUpdateProcessorFactoryTest, TestNoOpRegenerator, TestAuthorizationFramework, TestLuceneMatchVersion, DistributedFacetPivotSmallAdvancedTest, CoreAdminRequestStatusTest, TestQueryTypes, TestElisionMultitermQuery, DeleteStatusTest, SortSpecParsingTest, JsonLoaderTest, SpellPossibilityIteratorTest, ConfigureRecoveryStrategyTest, StatelessScriptUpdateProcessorFactoryTest, CursorMarkTest, TestTermsQParserPlugin, TestFieldResource, Tagger2Test, TestRandomFaceting, TestNumericTerms32, TestCloudInspectUtil, TestInPlaceUpdatesDistrib, TestSchemaManager, DistributedQueryComponentOptimizationTest, BasicAuthOnSingleNodeTest, TestBlendedInfixSuggestions, TestSimScenario] [junit4] Completed [237/913 (1!)] on J2 in 3.30s, 4 tests, 1 error <<< FAILURES! [...truncated 54790 lines...] [repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/8862/consoleText [repro] Revision: 0728ef06e98cee5a278b8d75054d0f0c9d33a5ac [repro] Ant options: "-Dargs=-XX:+UseCompressedOops -XX:+UseSerialGC" [repro] JUnit rest result XML files will be moved to: ./repro-reports [repro] ant clean [...truncated 6 lines...] [repro] Test suites by module: [repro] solr\core [repro] TestSimScenario [repro] ant compile-test [...truncated 3156 lines...] [repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.TestSimScenario" -Dtests.showOutput=onerror "-Dargs=-XX:+UseCompressedOops -XX:+UseSerialGC" -Dtests.seed=EA19F5D72A768827 -Dtests.slow=true -Dtests.locale=fr-MR -Dtests.timezone=Antarctica/McMurdo -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [...truncated 97 lines...] [repro] Failures w/original seeds: [repro] 0/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimScenario [repro] Exiting with code 0 [...truncated 78 lines...]
--------------------------------------------------------------------- To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org For additional commands, e-mail: builds-h...@lucene.apache.org