[
https://issues.apache.org/jira/browse/SOLR-12709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634477#comment-16634477
]
Steve Rowe commented on SOLR-12709:
-----------------------------------
{{TestSimExtremeIndexing.testScaleUp()}} failure from
[https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/1135/], the seed
only reproduced 1/5 iterations for me locally:
{noformat}
Checking out Revision 2369c8963412773592098475bdd8af1da81e3ac5
(refs/remotes/origin/master)
[...]
[smoker] [junit4] 2> 1221947 DEBUG
(TEST-TestSimExtremeIndexing.testScaleUp-seed#[4D17B79C7402215C]) [ ]
o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 218
[smoker] [junit4] 2> 1222001 INFO
(TEST-TestSimExtremeIndexing.testScaleUp-seed#[4D17B79C7402215C]) [ ]
o.a.s.c.a.s.SimSolrCloudTestCase #######################################
[smoker] [junit4] 2> ############ CLUSTER STATE ############
[smoker] [junit4] 2> #######################################
[smoker] [junit4] 2> ## Live nodes: 200
[smoker] [junit4] 2> ## Empty nodes: 191
[smoker] [junit4] 2> ## Dead nodes: 0
[smoker] [junit4] 2> ## Collections:
[smoker] [junit4] 2> ## * .system
[smoker] [junit4] 2> ## shardsTotal 1
[smoker] [junit4] 2> ## shardsState {active=1}
[smoker] [junit4] 2> ## shardsWithoutLeader 0
[smoker] [junit4] 2> ## totalReplicas 1
[smoker] [junit4] 2> ## activeReplicas 1
[smoker] [junit4] 2> ## inactiveReplicas 0
[smoker] [junit4] 2> ## totalActiveDocs 34
[smoker] [junit4] 2> ## bufferedDocs 0
[smoker] [junit4] 2> ## maxActiveSliceDocs 34
[smoker] [junit4] 2> ## minActiveSliceDocs 34
[smoker] [junit4] 2> ## avgActiveSliceDocs 34
[smoker] [junit4] 2> ## totalInactiveDocs 0
[smoker] [junit4] 2> ## maxInactiveSliceDocs 0
[smoker] [junit4] 2> ## minInactiveSliceDocs 0
[smoker] [junit4] 2> ## avgInactiveSliceDocs NaN
[smoker] [junit4] 2> ## totalActiveBytes 27,240
[smoker] [junit4] 2> ## maxActiveSliceBytes 27,240
[smoker] [junit4] 2> ## minActiveSliceBytes 27,240
[smoker] [junit4] 2> ## avgActiveSliceBytes 27,240
[smoker] [junit4] 2> ## totalInactiveBytes 0
[smoker] [junit4] 2> ## maxInactiveSliceBytes 0
[smoker] [junit4] 2> ## minInactiveSliceBytes 0
[smoker] [junit4] 2> ## avgInactiveSliceBytes NaN
[smoker] [junit4] 2> ## totalActiveDeletedDocs 0
[smoker] [junit4] 2> ## * testScaleUp_collection
[smoker] [junit4] 2> ## shardsTotal 92
[smoker] [junit4] 2> ## shardsState {active=46, construction=2,
inactive=44}
[smoker] [junit4] 2> ## shardsWithoutLeader 0
[smoker] [junit4] 2> ## totalReplicas 184
[smoker] [junit4] 2> ## activeReplicas 92
[smoker] [junit4] 2> ## inactiveReplicas 92
[smoker] [junit4] 2> ## totalActiveDocs 1,000,000,000
[smoker] [junit4] 2> ## bufferedDocs 204,400,000
[smoker] [junit4] 2> ## maxActiveSliceDocs 45,600,000
[smoker] [junit4] 2> ## minActiveSliceDocs 15,625,000
[smoker] [junit4] 2> ## avgActiveSliceDocs 17,295,652
[smoker] [junit4] 2> ## totalInactiveDocs 1,634,193,750
[smoker] [junit4] 2> ## maxInactiveSliceDocs 58,650,000
[smoker] [junit4] 2> ## minInactiveSliceDocs 22,800,000
[smoker] [junit4] 2> ## avgInactiveSliceDocs 35,525,951
[smoker] [junit4] 2> ## totalActiveBytes 390,916,321,320
[smoker] [junit4] 2> ## maxActiveSliceBytes 19,875,010,240
[smoker] [junit4] 2> ## minActiveSliceBytes 7,673,447,740
[smoker] [junit4] 2> ## avgActiveSliceBytes 8,498,180,898
[smoker] [junit4] 2> ## totalInactiveBytes 787,600,706,560
[smoker] [junit4] 2> ## maxInactiveSliceBytes 28,125,010,240
[smoker] [junit4] 2> ## minInactiveSliceBytes 11,400,010,240
[smoker] [junit4] 2> ## avgInactiveSliceBytes 17,121,754,490
[smoker] [junit4] 2> ## totalActiveDeletedDocs 0
[smoker] [junit4] 2> ## Max replicas per node: 45
[smoker] [junit4] 2> ## Min replicas per node: 0
[smoker] [junit4] 2> ## Total replicas: 185
[smoker] [junit4] 2> ## * .system 1
[smoker] [junit4] 2> ## - active 1
[smoker] [junit4] 2> ## * testScaleUp_collection 184
[smoker] [junit4] 2> ## - active 184
[smoker] [junit4] 2> ######### Solr op counts ##########
[smoker] [junit4] 2> ## - CREATE 1
[smoker] [junit4] 2> ## - DELETESTATUS 51
[smoker] [junit4] 2> ## - REQUESTSTATUS 51
[smoker] [junit4] 2> ## - SPLITSHARD 52
[smoker] [junit4] 2> ## - autoscaling 1
[smoker] [junit4] 2> ## - query 1
[smoker] [junit4] 2> ## - update 5042
[smoker] [junit4] 2> ######### Autoscaling event counts ###########
[smoker] [junit4] 2> ## * Trigger: scaleUpTrigger
[smoker] [junit4] 2> ## - FAILED 1
[smoker] [junit4] 2> ## - STARTED 8
[smoker] [junit4] 2> ## - SUCCEEDED 1
[smoker] [junit4] 2>
[smoker] [junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestSimExtremeIndexing -Dtests.method=testScaleUp
-Dtests.seed=4D17B79C7402215C -Dtests.multiplier=2 -Dtests.locale=ro
-Dtests.timezone=America/Catamarca -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[smoker] [junit4] FAILURE 4.88s J1 | TestSimExtremeIndexing.testScaleUp
<<<
[smoker] [junit4] > Throwable #1: java.lang.AssertionError:
{numFound=795600000,start=0,docs=[]} expected:<1000000000> but was:<795600000>
[smoker] [junit4] > at
__randomizedtesting.SeedInfo.seed([4D17B79C7402215C:6C49F13E782CFFFD]:0)
[smoker] [junit4] > at
org.apache.solr.cloud.autoscaling.sim.TestSimExtremeIndexing.testScaleUp(TestSimExtremeIndexing.java:121)
[smoker] [junit4] > at java.lang.Thread.run(Thread.java:748)
[smoker] [junit4] 2> 1222003 DEBUG
(SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [ ]
o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
[smoker] [junit4] 2> 1222003 DEBUG
(SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [ ]
o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[smoker] [junit4] 2> 1222003 WARN
(SolrRrdBackendFactory-5969-thread-1) [ ] o.a.s.m.r.SolrRrdBackendFactory
Error updating RRD data for solr.jvm
[smoker] [junit4] 2> java.io.IOException: java.lang.InterruptedException
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173) ~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138) ~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$maybeSyncBackends$5(SolrRrdBackendFactory.java:357)
~[java/:?]
[smoker] [junit4] 2> at java.util.HashMap.forEach(HashMap.java:1289)
~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.metrics.rrd.SolrRrdBackendFactory.maybeSyncBackends(SolrRrdBackendFactory.java:350)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$new$0(SolrRrdBackendFactory.java:110)
~[java/:?]
[smoker] [junit4] 2> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_172]
[smoker] [junit4] 2> at java.lang.Thread.run(Thread.java:748)
[?:1.8.0_172]
[smoker] [junit4] 2> Caused by: java.lang.InterruptedException
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651)
~[test/:?]
[smoker] [junit4] 2> ... 15 more
[smoker] [junit4] 2> 1222004 WARN (Simulated
OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread
OverseerTriggerThread woken up but we are closed, exiting.
[smoker] [junit4] 2> 1222004 DEBUG (simCloudManagerPool-5971-thread-33)
[ ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
[smoker] [junit4] 2> 1222005 WARN
(SolrRrdBackendFactory-5969-thread-1) [ ] o.a.s.m.r.SolrRrdBackendFactory
Error updating RRD data for solr.collection.testScaleUp_collection
[smoker] [junit4] 2> java.io.IOException:
java.util.concurrent.ExecutionException: java.io.IOException: already closed
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173) ~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138) ~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$maybeSyncBackends$5(SolrRrdBackendFactory.java:357)
~[java/:?]
[smoker] [junit4] 2> at java.util.HashMap.forEach(HashMap.java:1289)
~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.metrics.rrd.SolrRrdBackendFactory.maybeSyncBackends(SolrRrdBackendFactory.java:350)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$new$0(SolrRrdBackendFactory.java:110)
~[java/:?]
[smoker] [junit4] 2> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_172]
[smoker] [junit4] 2> at java.lang.Thread.run(Thread.java:748)
[?:1.8.0_172]
[smoker] [junit4] 2> Caused by:
java.util.concurrent.ExecutionException: java.io.IOException: already closed
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651)
~[test/:?]
[smoker] [junit4] 2> ... 15 more
[smoker] [junit4] 2> Caused by: java.io.IOException: already closed
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.ensureNotClosed(SimClusterStateProvider.java:2138)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simUpdate(SimClusterStateProvider.java:1367)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:738)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:650)
~[test/:?]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
~[java/:?]
[smoker] [junit4] 2> ... 3 more
[smoker] [junit4] 2> 1222006 WARN
(AutoscalingActionExecutor-5973-thread-1) [ ] o.a.s.c.a.SystemLogListener
Exception sending event to collection .system
[smoker] [junit4] 2> java.io.IOException:
java.util.concurrent.ExecutionException: java.io.IOException: already closed
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.SystemLogListener.onEvent(SystemLogListener.java:118)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:792)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:310)
~[java/:?]
[smoker] [junit4] 2> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
~[java/:?]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_172]
[smoker] [junit4] 2> at java.lang.Thread.run(Thread.java:748)
[?:1.8.0_172]
[smoker] [junit4] 2> Caused by:
java.util.concurrent.ExecutionException: java.io.IOException: already closed
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651)
~[test/:?]
[smoker] [junit4] 2> ... 9 more
[smoker] [junit4] 2> Caused by: java.io.IOException: already closed
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.ensureNotClosed(SimClusterStateProvider.java:2138)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simUpdate(SimClusterStateProvider.java:1367)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:738)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:650)
~[test/:?]
[smoker] [junit4] 2> ... 5 more
[smoker] [junit4] 2> 1222009 DEBUG (simCloudManagerPool-5971-thread-33)
[ ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 219
[smoker] [junit4] 2> 1222009 WARN
(AutoscalingActionExecutor-5973-thread-1) [ ] o.a.s.c.a.ScheduledTriggers
Exception executing actions
[smoker] [junit4] 2>
org.apache.solr.cloud.autoscaling.TriggerActionException: Error processing
action for trigger event: {
[smoker] [junit4] 2> "id":"1310b6c3b09c3cT31vg5ry49ycvekjxnv9ajjpzp",
[smoker] [junit4] 2> "source":"scaleUpTrigger",
[smoker] [junit4] 2> "eventTime":5366401710726204,
[smoker] [junit4] 2> "eventType":"INDEXSIZE",
[smoker] [junit4] 2> "properties":{
[smoker] [junit4] 2> "__start__":25,
[smoker] [junit4] 2> "aboveSize":{
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_0_0_0_replica_n217":"docs=24712500,
bytes=12075020480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_0_0_1_replica_n223":"docs=24712500,
bytes=12075010240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_0_1_0_replica_n127":"docs=24712500,
bytes=12015645480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_0_1_1_replica_n133":"docs=24712500,
bytes=12015635240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_1_0_0_replica_n199":"docs=24712500,
bytes=12065645480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_1_0_1_replica_n203":"docs=24712500,
bytes=12065635240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_1_1_0_replica_n179":"docs=24712500,
bytes=12103145480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_1_1_1_replica_n183":"docs=24712500,
bytes=12103135240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_1_replica_n11":"docs=45600000,
bytes=19875010240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_0_0_0_replica_n137":"docs=24712500,
bytes=11990645480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_0_0_1_replica_n143":"docs=24712500,
bytes=11990635240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_0_1_0_replica_n237":"docs=24712500,
bytes=12075020480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_0_1_1_replica_n243":"docs=24712500,
bytes=12075010240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_1_0_0_replica_n227":"docs=24712500,
bytes=12081270480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_1_0_1_replica_n233":"docs=24712500,
bytes=12081260240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_1_1_0_replica_n167":"docs=24712500,
bytes=12053145480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_1_1_1_replica_n173":"docs=24712500,
bytes=12053135240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_0_0_0_replica_n147":"docs=24712500,
bytes=12003145480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_0_0_1_replica_n153":"docs=24712500,
bytes=12003135240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_0_1_0_replica_n209":"docs=24712500,
bytes=12156270480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_0_1_1_replica_n213":"docs=24712500,
bytes=12156260240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_1_0_0_replica_n189":"docs=24712500,
bytes=12200020480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_1_0_1_replica_n193":"docs=24712500,
bytes=12200010240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_1_1_0_replica_n157":"docs=24712500,
bytes=12068770480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_1_1_1_replica_n163":"docs=24712500,
bytes=12068760240"},
[smoker] [junit4] 2> "belowSize":{},
[smoker] [junit4] 2> "_enqueue_time_":5366407403469004,
[smoker] [junit4] 2> "requestedOps":[
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_1_1"}]}}]}}
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:311)
~[java/:?]
[smoker] [junit4] 2> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
~[java/:?]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_172]
[smoker] [junit4] 2> at java.lang.Thread.run(Thread.java:748)
[?:1.8.0_172]
[smoker] [junit4] 2> Caused by: org.apache.solr.common.SolrException:
Unexpected exception while processing event: {
[smoker] [junit4] 2> "id":"1310b6c3b09c3cT31vg5ry49ycvekjxnv9ajjpzp",
[smoker] [junit4] 2> "source":"scaleUpTrigger",
[smoker] [junit4] 2> "eventTime":5366401710726204,
[smoker] [junit4] 2> "eventType":"INDEXSIZE",
[smoker] [junit4] 2> "properties":{
[smoker] [junit4] 2> "__start__":25,
[smoker] [junit4] 2> "aboveSize":{
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_0_0_0_replica_n217":"docs=24712500,
bytes=12075020480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_0_0_1_replica_n223":"docs=24712500,
bytes=12075010240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_0_1_0_replica_n127":"docs=24712500,
bytes=12015645480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_0_1_1_replica_n133":"docs=24712500,
bytes=12015635240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_1_0_0_replica_n199":"docs=24712500,
bytes=12065645480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_1_0_1_replica_n203":"docs=24712500,
bytes=12065635240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_1_1_0_replica_n179":"docs=24712500,
bytes=12103145480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_0_1_1_1_replica_n183":"docs=24712500,
bytes=12103135240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard1_1_replica_n11":"docs=45600000,
bytes=19875010240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_0_0_0_replica_n137":"docs=24712500,
bytes=11990645480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_0_0_1_replica_n143":"docs=24712500,
bytes=11990635240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_0_1_0_replica_n237":"docs=24712500,
bytes=12075020480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_0_1_1_replica_n243":"docs=24712500,
bytes=12075010240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_1_0_0_replica_n227":"docs=24712500,
bytes=12081270480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_1_0_1_replica_n233":"docs=24712500,
bytes=12081260240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_1_1_0_replica_n167":"docs=24712500,
bytes=12053145480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_0_1_1_1_replica_n173":"docs=24712500,
bytes=12053135240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_0_0_0_replica_n147":"docs=24712500,
bytes=12003145480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_0_0_1_replica_n153":"docs=24712500,
bytes=12003135240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_0_1_0_replica_n209":"docs=24712500,
bytes=12156270480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_0_1_1_replica_n213":"docs=24712500,
bytes=12156260240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_1_0_0_replica_n189":"docs=24712500,
bytes=12200020480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_1_0_1_replica_n193":"docs=24712500,
bytes=12200010240",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_1_1_0_replica_n157":"docs=24712500,
bytes=12068770480",
[smoker] [junit4] 2>
"testScaleUp_collection_shard2_1_1_1_1_replica_n163":"docs=24712500,
bytes=12068760240"},
[smoker] [junit4] 2> "belowSize":{},
[smoker] [junit4] 2> "_enqueue_time_":5366407403469004,
[smoker] [junit4] 2> "requestedOps":[
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_1_1"}]}}]}}
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:120)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308)
~[java/:?]
[smoker] [junit4] 2> ... 6 more
[smoker] [junit4] 2> Caused by: org.apache.solr.common.SolrException:
Unexpected exception executing operation:
action=SPLITSHARD&async=scaleUpTrigger/1310b6c3b09c3cT31vg5ry49ycvekjxnv9ajjpzp/0&waitForFinalState=true&collection=testScaleUp_collection&shard=shard2_0_1_0_1
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:110)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308)
~[java/:?]
[smoker] [junit4] 2> ... 6 more
[smoker] [junit4] 2> Caused by: java.io.IOException:
java.lang.InterruptedException
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:80)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308)
~[java/:?]
[smoker] [junit4] 2> ... 6 more
[smoker] [junit4] 2> Caused by: java.lang.InterruptedException
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:80)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308)
~[java/:?]
[smoker] [junit4] 2> ... 6 more
[smoker] [junit4] 2> 1222010 DEBUG
(AutoscalingActionExecutor-5973-thread-1) [ ] o.a.s.c.a.ScheduledTriggers --
processing took 176888 ms for event id=1310b6c3b09c3cT31vg5ry49ycvekjxnv9ajjpzp
[smoker] [junit4] 2> 1222007 DEBUG
(SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [ ]
o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[smoker] [junit4] 2> 1222010 DEBUG
(SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [ ]
o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool
executor
[smoker] [junit4] 2> 1222010 DEBUG
(SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [ ]
o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[smoker] [junit4] 2> 1222011 DEBUG
(SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [ ]
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[smoker] [junit4] 2> 1222011 WARN
(SolrRrdBackendFactory-5969-thread-1) [ ] o.a.s.m.r.SolrRrdBackendFactory
Error updating RRD data for solr.node
[smoker] [junit4] 2> java.io.IOException:
java.util.concurrent.ExecutionException: java.io.IOException: already closed
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173) ~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138) ~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$maybeSyncBackends$5(SolrRrdBackendFactory.java:357)
~[java/:?]
[smoker] [junit4] 2> at java.util.HashMap.forEach(HashMap.java:1289)
~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.metrics.rrd.SolrRrdBackendFactory.maybeSyncBackends(SolrRrdBackendFactory.java:350)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$new$0(SolrRrdBackendFactory.java:110)
~[java/:?]
[smoker] [junit4] 2> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_172]
[smoker] [junit4] 2> at java.lang.Thread.run(Thread.java:748)
[?:1.8.0_172]
[smoker] [junit4] 2> Caused by:
java.util.concurrent.ExecutionException: java.io.IOException: already closed
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651)
~[test/:?]
[smoker] [junit4] 2> ... 15 more
[smoker] [junit4] 2> Caused by: java.io.IOException: already closed
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.ensureNotClosed(SimClusterStateProvider.java:2138)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simUpdate(SimClusterStateProvider.java:1367)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:738)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:650)
~[test/:?]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
~[java/:?]
[smoker] [junit4] 2> ... 3 more
[smoker] [junit4] 2> NOTE: leaving temporary files on disk at:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-8.0.0/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimExtremeIndexing_4D17B79C7402215C-001
[smoker] [junit4] 2> 1222029 WARN
(SolrRrdBackendFactory-5969-thread-1) [ ] o.a.s.m.r.SolrRrdBackendFactory
Error sending RRD data updates
[smoker] [junit4] 2> java.io.IOException:
java.util.concurrent.RejectedExecutionException: Task
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$129/198524200@1817d8df
rejected from
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@2a463d6b[Shutting
down, pool size = 142, active threads = 0, queued tasks = 0, completed tasks =
5300]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484) ~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:448) ~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.metrics.rrd.SolrRrdBackendFactory.maybeSyncBackends(SolrRrdBackendFactory.java:366)
~[java/:?]
[smoker] [junit4] 2> at
org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$new$0(SolrRrdBackendFactory.java:110)
~[java/:?]
[smoker] [junit4] 2> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_172]
[smoker] [junit4] 2> at java.lang.Thread.run(Thread.java:748)
[?:1.8.0_172]
[smoker] [junit4] 2> Caused by:
java.util.concurrent.RejectedExecutionException: Task
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$129/198524200@1817d8df
rejected from
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@2a463d6b[Shutting
down, pool size = 142, active threads = 0, queued tasks = 0, completed tasks =
5300]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
~[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
~[?:1.8.0_172]
[smoker] [junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:194)
~[java/:?]
[smoker] [junit4] 2> at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
~[?:1.8.0_172]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.submit(SimCloudManager.java:569)
~[test/:?]
[smoker] [junit4] 2> at
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:650)
~[test/:?]
[smoker] [junit4] 2> ... 13 more
[smoker] [junit4] 2> Sep 27, 2018 7:42:57 PM
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[smoker] [junit4] 2> WARNING: Will linger awaiting termination of 18
leaked thread(s).
[smoker] [junit4] 2> NOTE: test params are: codec=Asserting(Lucene80),
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2a3300f8),
locale=ro, timezone=America/Catamarca
[smoker] [junit4] 2> NOTE: Linux 4.4.0-130-generic amd64/Oracle
Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=223113456,total=518520832
[smoker] [junit4] 2> NOTE: All tests run in this JVM:
[SearchHandlerTest, AutoScalingHandlerTest, TestRangeQuery,
TestPerFieldSimilarity, TestBlobHandler, TestManagedSchema,
ParsingFieldUpdateProcessorsTest, MetricsHistoryHandlerTest, TestLegacyTerms,
DirectSolrSpellCheckerTest, ConfigSetsAPITest, TestConfig,
TestNumericRangeQuery64, TestQueryUtils, TestValueSourceCache,
SolrInfoBeanTest, TestMiniSolrCloudClusterSSL, SolrIndexSplitterTest,
ProtectedTermFilterFactoryTest, DefaultValueUpdateProcessorTest,
TestClusterProperties, TestAuthorizationFramework,
TriggerSetPropertiesIntegrationTest, TestMacroExpander, TestSSLRandomization,
TestDistribDocBasedVersion, FacetPivotSmallTest, HLLUtilTest,
CdcrRequestHandlerTest, TestWithCollection, DateRangeFieldTest,
V2StandaloneTest, TestCoreDiscovery, BasicFunctionalityTest,
TestSolrIndexConfig, CachingDirectoryFactoryTest, NodeMarkersRegistrationTest,
TestReload, DeleteStatusTest, DisMaxRequestHandlerTest,
TestJavabinTupleStreamParser, TestManagedResource, TestCustomDocTransformer,
TestCloudJSONFacetSKG, AnalysisErrorHandlingTest, JavabinLoaderTest,
TestReversedWildcardFilterFactory, TestSimNodeAddedTrigger,
SimpleMLTQParserTest, HealthCheckHandlerTest, JvmMetricsTest,
TestBlendedInfixSuggestions, TestLMDirichletSimilarityFactory, TestFaceting,
TestDirectoryFactory, HdfsRecoverLeaseTest, TestExceedMaxTermLength,
TestDynamicFieldResource, TestNRTOpen, TestHdfsCloudBackupRestore,
SpellCheckCollatorTest, TestUpdate, TestTolerantSearch, UtilsToolTest,
TestFastLRUCache, BasicAuthIntegrationTest, TestOmitPositions,
QueryElevationComponentTest, TestDistributedSearch,
TestPayloadScoreQParserPlugin, PeerSyncTest, SyncSliceTest, HdfsDirectoryTest,
ActionThrottleTest, DistributedSpellCheckComponentTest,
HdfsChaosMonkeyNothingIsSafeTest, TriggerCooldownIntegrationTest,
LukeRequestHandlerTest, TestNumericTerms64, TestImplicitCoreProperties,
LargeFieldTest, TestDistributedStatsComponentCardinality,
DistributedQueryComponentCustomSortTest, TestJsonFacetRefinement,
WrapperMergePolicyFactoryTest, CdcrBootstrapTest,
TestExclusionRuleCollectionAccess, PeerSyncReplicationTest,
BasicDistributedZkTest, ImplicitSnitchTest, DirectUpdateHandlerTest,
TestClusterStateMutator, CollectionsAPIAsyncDistributedZkTest,
LeaderFailureAfterFreshStartTest, ScriptEngineTest,
DistributedFacetSimpleRefinementLongTailTest, SimplePostToolTest,
TestSolr4Spatial, SuggesterFSTTest, SchemaApiFailureTest, TestNoOpRegenerator,
DirectoryFactoryTest, TestBulkSchemaConcurrent,
ChaosMonkeyNothingIsSafeWithPullReplicasTest, OpenCloseCoreStressTest,
ReplicationFactorTest, TestSolrCloudWithHadoopAuthPlugin,
TestReplicationHandlerBackup, TestUseDocValuesAsStored2,
DistributedDebugComponentTest, TestTestInjection, MultiTermTest,
ZkNodePropsTest, NodeAddedTriggerTest, Tagger2Test, DocValuesNotIndexedTest,
TestMinMaxOnMultiValuedField, QueryParsingTest, TestSolrQueryResponse,
FieldMutatingUpdateProcessorTest, AssignBackwardCompatibilityTest,
TermVectorComponentDistributedTest, TestFieldTypeCollectionResource,
TestUnifiedSolrHighlighter, StatsComponentTest, TestSort, TestCollationField,
TestConfigSetsAPIZkFailure, TestSimDistribStateManager, PreAnalyzedFieldTest,
BadComponentTest, NumberUtilsTest, SpatialRPTFieldTypeTest,
TestSolrCloudSnapshots, DeleteInactiveReplicaTest, DeleteNodeTest,
DistribJoinFromCollectionTest, DistributedQueueTest,
DistributedVersionInfoTest, LIROnShardRestartTest, LIRRollingUpdatesTest,
MoveReplicaTest, OverseerStatusTest, RecoveryZkTest, RemoteQueryErrorTest,
ReplaceNodeNoTargetTest, SSLMigrationTest, SaslZkACLProviderTest,
ShardRoutingTest, SharedFSAutoReplicaFailoverTest, SolrCLIZkUtilsTest,
SolrCloudExampleTest, SolrXmlInZkTest, TestCloudConsistency,
TestCloudDeleteByQuery, TestCloudPseudoReturnFields,
TestDeleteCollectionOnDownNodes, TestLeaderElectionWithEmptyReplica,
TestLeaderElectionZkExpiry, TestLeaderInitiatedRecoveryThread, TestPullReplica,
TestPullReplicaErrorHandling, CollectionReloadTest, CustomCollectionTest,
AutoAddReplicasPlanActionTest, ExecutePlanActionTest,
MetricTriggerIntegrationTest, ScheduledTriggerTest,
SearchRateTriggerIntegrationTest, SystemLogListenerTest, TestSimExtremeIndexing]
[smoker] [junit4] Completed [539/835 (1!)] on J1 in 5.23s, 1 test, 1
failure <<< FAILURES!
{noformat}
> Simulate a 1 bln docs scaling-up scenario
> -----------------------------------------
>
> Key: SOLR-12709
> URL: https://issues.apache.org/jira/browse/SOLR-12709
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Priority: Major
> Fix For: 7.5, master (8.0)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
