Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4533/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaOnIndexing

Error Message:
Time out waiting for LIR state get removed

Stack Trace:
java.util.concurrent.TimeoutException: Time out waiting for LIR state get 
removed
        at 
__randomizedtesting.SeedInfo.seed([5B3BA86FA862E0EC:224084E00BFF2ACF]:0)
        at org.apache.solr.util.TimeOut.waitFor(TimeOut.java:66)
        at 
org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaOnIndexing(DeleteReplicaTest.java:331)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1846 lines...]
   [junit4] JVM J1: stdout was not empty, see: 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/core/test/temp/junit4-J1-20180328_232802_7198642311986570932812.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] codec: DummyCompressingStoredFields, pf: FST50, dvf: Asserting
   [junit4] <<< JVM J1: EOF ----

[...truncated 12015 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> Creating dataDir: 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/init-core-data-001
   [junit4]   2> 2854132 WARN  
(SUITE-DeleteReplicaTest-seed#[5B3BA86FA862E0EC]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=10 numCloses=10
   [junit4]   2> 2854132 INFO  
(SUITE-DeleteReplicaTest-seed#[5B3BA86FA862E0EC]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2854137 INFO  
(SUITE-DeleteReplicaTest-seed#[5B3BA86FA862E0EC]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) 
w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2854137 INFO  
(SUITE-DeleteReplicaTest-seed#[5B3BA86FA862E0EC]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2854138 INFO  
(SUITE-DeleteReplicaTest-seed#[5B3BA86FA862E0EC]-worker) [    ] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001
   [junit4]   2> 2854138 INFO  
(SUITE-DeleteReplicaTest-seed#[5B3BA86FA862E0EC]-worker) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2854139 INFO  (Thread-4143) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2854139 INFO  (Thread-4143) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 2854143 ERROR (Thread-4143) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2854243 INFO  
(SUITE-DeleteReplicaTest-seed#[5B3BA86FA862E0EC]-worker) [    ] 
o.a.s.c.ZkTestServer start zk server on port:62515
   [junit4]   2> 2854255 INFO  (zkConnectionManagerCallback-5316-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854262 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2854262 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2854263 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2854264 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2854264 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2854264 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 2854265 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2854269 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2854269 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2854269 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 2854270 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@7ae4c74a{/solr,null,AVAILABLE}
   [junit4]   2> 2854272 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@30516d13{HTTP/1.1,[http/1.1]}{127.0.0.1:62517}
   [junit4]   2> 2854272 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.e.j.s.Server Started @2854353ms
   [junit4]   2> 2854272 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=62517}
   [junit4]   2> 2854272 ERROR (jetty-launcher-5313-thread-3) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2854272 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2854273 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
8.0.0
   [junit4]   2> 2854273 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2854273 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2854273 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-03-29T00:36:39.100Z
   [junit4]   2> 2854275 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2854275 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2854275 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 2854285 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@3d1f676f{/solr,null,AVAILABLE}
   [junit4]   2> 2854285 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@641de65e{HTTP/1.1,[http/1.1]}{127.0.0.1:62518}
   [junit4]   2> 2854285 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.e.j.s.Server Started @2854366ms
   [junit4]   2> 2854285 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=62518}
   [junit4]   2> 2854286 ERROR (jetty-launcher-5313-thread-1) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2854286 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2854286 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
8.0.0
   [junit4]   2> 2854286 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2854286 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2854287 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-03-29T00:36:39.113Z
   [junit4]   2> 2854288 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@6bedeadb{/solr,null,AVAILABLE}
   [junit4]   2> 2854289 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@70a2355c{HTTP/1.1,[http/1.1]}{127.0.0.1:62519}
   [junit4]   2> 2854289 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.e.j.s.Server Started @2854370ms
   [junit4]   2> 2854289 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=62519}
   [junit4]   2> 2854289 ERROR (jetty-launcher-5313-thread-2) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2854289 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2854290 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
8.0.0
   [junit4]   2> 2854290 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2854290 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2854290 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-03-29T00:36:39.117Z
   [junit4]   2> 2854294 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2854294 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2854294 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 2854297 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@76d387f5{/solr,null,AVAILABLE}
   [junit4]   2> 2854297 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@3626333b{HTTP/1.1,[http/1.1]}{127.0.0.1:62521}
   [junit4]   2> 2854297 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.e.j.s.Server Started @2854378ms
   [junit4]   2> 2854298 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=62521}
   [junit4]   2> 2854298 ERROR (jetty-launcher-5313-thread-4) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2854298 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2854299 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
8.0.0
   [junit4]   2> 2854299 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2854299 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2854299 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-03-29T00:36:39.126Z
   [junit4]   2> 2854300 INFO  (zkConnectionManagerCallback-5318-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854303 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2854308 INFO  (zkConnectionManagerCallback-5320-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854315 INFO  (zkConnectionManagerCallback-5322-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854316 INFO  (zkConnectionManagerCallback-5324-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854316 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2854317 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2854318 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2854327 INFO  (jetty-launcher-5313-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:62515/solr
   [junit4]   2> 2854327 INFO  (jetty-launcher-5313-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:62515/solr
   [junit4]   2> 2854331 INFO  (jetty-launcher-5313-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:62515/solr
   [junit4]   2> 2854332 INFO  (jetty-launcher-5313-thread-4) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:62515/solr
   [junit4]   2> 2854333 INFO  (zkConnectionManagerCallback-5330-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854334 INFO  (zkConnectionManagerCallback-5332-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854334 INFO  (zkConnectionManagerCallback-5337-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854335 INFO  (zkConnectionManagerCallback-5340-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854340 INFO  (zkConnectionManagerCallback-5342-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854341 INFO  (zkConnectionManagerCallback-5344-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854342 INFO  (zkConnectionManagerCallback-5346-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854342 INFO  (zkConnectionManagerCallback-5348-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854611 INFO  (jetty-launcher-5313-thread-1) 
[n:127.0.0.1:62518_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2854611 INFO  (jetty-launcher-5313-thread-2) 
[n:127.0.0.1:62519_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2854612 INFO  (jetty-launcher-5313-thread-1) 
[n:127.0.0.1:62518_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:62518_solr
   [junit4]   2> 2854613 INFO  (jetty-launcher-5313-thread-3) 
[n:127.0.0.1:62517_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2854613 INFO  (jetty-launcher-5313-thread-2) 
[n:127.0.0.1:62519_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:62519_solr
   [junit4]   2> 2854613 INFO  (jetty-launcher-5313-thread-1) 
[n:127.0.0.1:62518_solr    ] o.a.s.c.Overseer Overseer 
(id=72140217251528713-127.0.0.1:62518_solr-n_0000000000) starting
   [junit4]   2> 2854616 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2854616 INFO  (zkCallback-5341-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2854617 INFO  (zkCallback-5347-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2854619 INFO  (jetty-launcher-5313-thread-3) 
[n:127.0.0.1:62517_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 2147483647 transient cores
   [junit4]   2> 2854620 INFO  (jetty-launcher-5313-thread-3) 
[n:127.0.0.1:62517_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:62517_solr
   [junit4]   2> 2854623 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2854623 INFO  (zkCallback-5341-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2854624 INFO  (zkCallback-5347-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2854632 INFO  (zkConnectionManagerCallback-5356-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2854635 INFO  (jetty-launcher-5313-thread-1) 
[n:127.0.0.1:62518_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 2854637 INFO  (jetty-launcher-5313-thread-1) 
[n:127.0.0.1:62518_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:62515/solr ready
   [junit4]   2> 2854650 INFO  (jetty-launcher-5313-thread-1) 
[n:127.0.0.1:62518_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 2147483647 transient cores
   [junit4]   2> 2854650 INFO  (jetty-launcher-5313-thread-1) 
[n:127.0.0.1:62518_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:62518_solr
   [junit4]   2> 2854657 INFO  (zkCallback-5355-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2854667 INFO  (zkCallback-5347-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2854667 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2854668 INFO  (zkCallback-5341-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2854689 INFO  (jetty-launcher-5313-thread-4) 
[n:127.0.0.1:62521_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (3)
   [junit4]   2> 2854692 INFO  (jetty-launcher-5313-thread-4) 
[n:127.0.0.1:62521_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2854695 INFO  (jetty-launcher-5313-thread-4) 
[n:127.0.0.1:62521_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 2147483647 transient cores
   [junit4]   2> 2854695 INFO  (jetty-launcher-5313-thread-4) 
[n:127.0.0.1:62521_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:62521_solr
   [junit4]   2> 2854697 INFO  (zkCallback-5347-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2854697 INFO  (zkCallback-5341-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2854697 INFO  (zkCallback-5355-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2854697 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2854699 INFO  (zkCallback-5345-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2855174 INFO  (jetty-launcher-5313-thread-1) 
[n:127.0.0.1:62518_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62518.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2855194 INFO  (jetty-launcher-5313-thread-1) 
[n:127.0.0.1:62518_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62518.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2855194 INFO  (jetty-launcher-5313-thread-1) 
[n:127.0.0.1:62518_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62518.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2855195 INFO  (jetty-launcher-5313-thread-1) 
[n:127.0.0.1:62518_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node1/.
   [junit4]   2> 2855228 INFO  (jetty-launcher-5313-thread-2) 
[n:127.0.0.1:62519_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62519.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2855236 INFO  (jetty-launcher-5313-thread-2) 
[n:127.0.0.1:62519_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62519.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2855236 INFO  (jetty-launcher-5313-thread-2) 
[n:127.0.0.1:62519_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62519.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2855237 INFO  (jetty-launcher-5313-thread-2) 
[n:127.0.0.1:62519_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node2/.
   [junit4]   2> 2855241 INFO  (zkConnectionManagerCallback-5360-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2855243 INFO  (jetty-launcher-5313-thread-2) 
[n:127.0.0.1:62519_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (4)
   [junit4]   2> 2855244 INFO  (jetty-launcher-5313-thread-2) 
[n:127.0.0.1:62519_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:62515/solr ready
   [junit4]   2> 2855296 INFO  (jetty-launcher-5313-thread-3) 
[n:127.0.0.1:62517_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62517.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2855307 INFO  (jetty-launcher-5313-thread-3) 
[n:127.0.0.1:62517_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62517.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2855307 INFO  (jetty-launcher-5313-thread-3) 
[n:127.0.0.1:62517_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62517.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2855308 INFO  (jetty-launcher-5313-thread-3) 
[n:127.0.0.1:62517_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node3/.
   [junit4]   2> 2855312 INFO  (zkConnectionManagerCallback-5364-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2855315 INFO  (jetty-launcher-5313-thread-3) 
[n:127.0.0.1:62517_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (4)
   [junit4]   2> 2855317 INFO  (jetty-launcher-5313-thread-3) 
[n:127.0.0.1:62517_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:62515/solr ready
   [junit4]   2> 2855497 INFO  (jetty-launcher-5313-thread-4) 
[n:127.0.0.1:62521_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62521.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2855505 INFO  (jetty-launcher-5313-thread-4) 
[n:127.0.0.1:62521_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62521.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2855506 INFO  (jetty-launcher-5313-thread-4) 
[n:127.0.0.1:62521_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62521.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2855507 INFO  (jetty-launcher-5313-thread-4) 
[n:127.0.0.1:62521_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node4/.
   [junit4]   2> 2855512 INFO  (zkConnectionManagerCallback-5368-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2855514 INFO  (jetty-launcher-5313-thread-4) 
[n:127.0.0.1:62521_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (4)
   [junit4]   2> 2855516 INFO  (jetty-launcher-5313-thread-4) 
[n:127.0.0.1:62521_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:62515/solr ready
   [junit4]   2> 2855563 INFO  (zkConnectionManagerCallback-5370-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2855568 INFO  (zkConnectionManagerCallback-5374-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2855571 INFO  
(SUITE-DeleteReplicaTest-seed#[5B3BA86FA862E0EC]-worker) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2855573 INFO  
(SUITE-DeleteReplicaTest-seed#[5B3BA86FA862E0EC]-worker) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:62515/solr ready
   [junit4]   2> 2855627 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting raceConditionOnDeleteAndRegisterReplica
   [junit4]   2> 2855631 INFO  (qtp1124586875-23549) [n:127.0.0.1:62521_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&collection.configName=conf&name=raceDeleteReplica&nrtReplicas=2&action=CREATE&numShards=1&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 2855634 INFO  (OverseerThreadFactory-9176-thread-1) [    ] 
o.a.s.c.a.c.CreateCollectionCmd Create collection raceDeleteReplica
   [junit4]   2> 2855757 INFO  
(OverseerStateUpdate-72140217251528713-127.0.0.1:62518_solr-n_0000000000) [    
] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"raceDeleteReplica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"raceDeleteReplica_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:62517/solr";,
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2855761 INFO  
(OverseerStateUpdate-72140217251528713-127.0.0.1:62518_solr-n_0000000000) [    
] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"raceDeleteReplica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"raceDeleteReplica_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:62519/solr";,
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2855980 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=raceDeleteReplica_shard1_replica_n2&action=CREATE&numShards=1&collection=raceDeleteReplica&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2855980 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=raceDeleteReplica_shard1_replica_n1&action=CREATE&numShards=1&collection=raceDeleteReplica&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2855982 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 2856098 INFO  (zkCallback-5347-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2856098 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2856098 INFO  (zkCallback-5347-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2856099 INFO  (zkCallback-5343-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2857007 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene 
MatchVersion: 8.0.0
   [junit4]   2> 2857007 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene 
MatchVersion: 8.0.0
   [junit4]   2> 2857012 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.s.IndexSchema 
[raceDeleteReplica_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 2857012 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.s.IndexSchema 
[raceDeleteReplica_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 2857016 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema 
minimal/1.1 with uniqueid field id
   [junit4]   2> 2857016 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 
'raceDeleteReplica_shard1_replica_n1' using configuration from collection 
raceDeleteReplica, trusted=true
   [junit4]   2> 2857016 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema 
minimal/1.1 with uniqueid field id
   [junit4]   2> 2857016 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 
'raceDeleteReplica_shard1_replica_n2' using configuration from collection 
raceDeleteReplica, trusted=true
   [junit4]   2> 2857017 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring 
for 'solr_62517.solr.core.raceDeleteReplica.shard1.replica_n1' (registry 
'solr.core.raceDeleteReplica.shard1.replica_n1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2857017 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 2857017 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.SolrCore 
[[raceDeleteReplica_shard1_replica_n1] ] Opening new SolrCore at 
[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node3/raceDeleteReplica_shard1_replica_n1],
 
dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node3/./raceDeleteReplica_shard1_replica_n1/data/]
   [junit4]   2> 2857017 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring 
for 'solr_62519.solr.core.raceDeleteReplica.shard1.replica_n2' (registry 
'solr.core.raceDeleteReplica.shard1.replica_n2') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2857017 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 2857017 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.SolrCore 
[[raceDeleteReplica_shard1_replica_n2] ] Opening new SolrCore at 
[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node2/raceDeleteReplica_shard1_replica_n2],
 
dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node2/./raceDeleteReplica_shard1_replica_n2/data/]
   [junit4]   2> 2857106 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2857106 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2857107 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 2857107 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 2857108 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@2dd651d9[raceDeleteReplica_shard1_replica_n1] main]
   [junit4]   2> 2857110 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2857110 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2857110 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2857110 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2857111 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will 
be reserved for 10000ms.
   [junit4]   2> 2857111 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1596230259900940288
   [junit4]   2> 2857111 INFO  
(searcherExecutor-9185-thread-1-processing-n:127.0.0.1:62517_solr 
x:raceDeleteReplica_shard1_replica_n1 c:raceDeleteReplica s:shard1 
r:core_node3) [n:127.0.0.1:62517_solr c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.SolrCore 
[raceDeleteReplica_shard1_replica_n1] Registered new searcher 
Searcher@2dd651d9[raceDeleteReplica_shard1_replica_n1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2857112 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 2857112 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 2857114 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@3322cd88[raceDeleteReplica_shard1_replica_n2] main]
   [junit4]   2> 2857116 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2857116 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2857117 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will 
be reserved for 10000ms.
   [junit4]   2> 2857117 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1596230259907231744
   [junit4]   2> 2857118 INFO  
(searcherExecutor-9186-thread-1-processing-n:127.0.0.1:62519_solr 
x:raceDeleteReplica_shard1_replica_n2 c:raceDeleteReplica s:shard1 
r:core_node4) [n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.SolrCore 
[raceDeleteReplica_shard1_replica_n2] Registered new searcher 
Searcher@3322cd88[raceDeleteReplica_shard1_replica_n2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2857118 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update 
of terms at /collections/raceDeleteReplica/terms/shard1 to 
Terms{values={core_node3=0}, version=0}
   [junit4]   2> 2857122 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update 
of terms at /collections/raceDeleteReplica/terms/shard1 to 
Terms{values={core_node3=0, core_node4=0}, version=1}
   [junit4]   2> 2857125 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext 
Enough replicas found to continue.
   [junit4]   2> 2857125 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may 
be the new leader - try and sync
   [junit4]   2> 2857125 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:62517/solr/raceDeleteReplica_shard1_replica_n1/
   [junit4]   2> 2857126 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: 
core=raceDeleteReplica_shard1_replica_n1 url=http://127.0.0.1:62517/solr START 
replicas=[http://127.0.0.1:62519/solr/raceDeleteReplica_shard1_replica_n2/] 
nUpdates=100
   [junit4]   2> 2857128 INFO  (qtp153355834-23540) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.S.Request 
[raceDeleteReplica_shard1_replica_n2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=1
   [junit4]   2> 2857228 INFO  (zkCallback-5343-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2857228 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2857228 INFO  (zkCallback-5347-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2857228 INFO  (zkCallback-5347-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2857433 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: 
core=raceDeleteReplica_shard1_replica_n1 url=http://127.0.0.1:62517/solr DONE.  
We have no versions.  sync failed.
   [junit4]   2> 2857433 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to 
sync with shard failed, moving to the next candidate
   [junit4]   2> 2857433 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We 
failed sync, but we have no versions - we can't sync in that case - we were 
active before, so become leader anyway
   [junit4]   2> 2857433 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found 
all replicas participating in election, clear LIR
   [junit4]   2> 2857440 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am 
the new leader: 
http://127.0.0.1:62517/solr/raceDeleteReplica_shard1_replica_n1/ shard1
   [junit4]   2> 2857543 INFO  (zkCallback-5347-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2857543 INFO  (zkCallback-5347-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2857544 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2857544 INFO  (zkCallback-5343-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2857595 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no 
recovery necessary
   [junit4]   2> 2857599 INFO  (qtp559027215-23551) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null 
path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=raceDeleteReplica_shard1_replica_n1&action=CREATE&numShards=1&collection=raceDeleteReplica&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=1618
   [junit4]   2> 2857708 INFO  (zkCallback-5343-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2857708 INFO  (zkCallback-5347-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2857708 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2857708 INFO  (zkCallback-5347-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2858134 INFO  (qtp153355834-23536) [n:127.0.0.1:62519_solr 
c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null 
path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=raceDeleteReplica_shard1_replica_n2&action=CREATE&numShards=1&collection=raceDeleteReplica&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=2154
   [junit4]   2> 2858137 INFO  (qtp1124586875-23549) [n:127.0.0.1:62521_solr    
] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 2858234 INFO  (zkCallback-5347-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2858234 INFO  (zkCallback-5347-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2858234 INFO  (zkCallback-5343-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2858234 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2859139 INFO  (qtp1124586875-23549) [n:127.0.0.1:62521_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=2&collection.configName=conf&name=raceDeleteReplica&nrtReplicas=2&action=CREATE&numShards=1&wt=javabin&version=2}
 status=0 QTime=3508
   [junit4]   2> 2859144 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.e.j.s.AbstractConnector Stopped 
ServerConnector@70a2355c{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2859145 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=61884269
   [junit4]   2> 2859145 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.node, tag=null
   [junit4]   2> 2859145 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@6be753bb: rootName = 
solr_62519, domain = solr.node, service url = null, agent id = null] for 
registry solr.node / com.codahale.metrics.MetricRegistry@523283aa
   [junit4]   2> 2859152 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.jvm, tag=null
   [junit4]   2> 2859152 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@5ea6704d: rootName = 
solr_62519, domain = solr.jvm, service url = null, agent id = null] for 
registry solr.jvm / com.codahale.metrics.MetricRegistry@6d9ec671
   [junit4]   2> 2859156 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.jetty, tag=null
   [junit4]   2> 2859156 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@60649ec7: rootName = 
solr_62519, domain = solr.jetty, service url = null, agent id = null] for 
registry solr.jetty / com.codahale.metrics.MetricRegistry@54ab1d93
   [junit4]   2> 2859158 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.c.ZkController Remove node as live in 
ZooKeeper:/live_nodes/127.0.0.1:62519_solr
   [junit4]   2> 2859159 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.cluster, tag=null
   [junit4]   2> 2859161 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 2859161 INFO  (zkCallback-5359-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 2859162 INFO  (coreCloseExecutor-9195-thread-1) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.SolrCore 
[raceDeleteReplica_shard1_replica_n2]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@5fa0d1a4
   [junit4]   2> 2859162 INFO  (coreCloseExecutor-9195-thread-1) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.core.raceDeleteReplica.shard1.replica_n2, 
tag=5fa0d1a4
   [junit4]   2> 2859162 INFO  (coreCloseExecutor-9195-thread-1) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter Closing 
reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@23b7024b: rootName 
= solr_62519, domain = solr.core.raceDeleteReplica.shard1.replica_n2, service 
url = null, agent id = null] for registry 
solr.core.raceDeleteReplica.shard1.replica_n2 / 
com.codahale.metrics.MetricRegistry@5bc8ed2f
   [junit4]   2> 2859162 INFO  (zkCallback-5363-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 2859162 INFO  (zkCallback-5347-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 2859162 INFO  (zkCallback-5373-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 2859162 INFO  (zkCallback-5355-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 2859162 INFO  (zkCallback-5341-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 2859162 INFO  (zkCallback-5367-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 2859162 INFO  (zkCallback-5345-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 2859188 INFO  (coreCloseExecutor-9195-thread-1) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.collection.raceDeleteReplica.shard1.leader, 
tag=5fa0d1a4
   [junit4]   2> 2859199 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@6bedeadb{/solr,null,UNAVAILABLE}
   [junit4]   2> 2859200 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.e.j.s.session Stopped scavenging
   [junit4]   2> 2859262 INFO  (zkCallback-5373-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2859262 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2859262 INFO  (zkCallback-5343-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2859262 INFO  (zkCallback-5373-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 2859268 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 2859272 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2859272 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2859273 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.e.j.s.session Scavenging every 660000ms
   [junit4]   2> 2859275 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@74ddc31e{/solr,null,AVAILABLE}
   [junit4]   2> 2859276 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.e.j.s.AbstractConnector Started 
ServerConnector@4971bfd5{HTTP/1.1,[http/1.1]}{127.0.0.1:62519}
   [junit4]   2> 2859276 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.e.j.s.Server Started @2859357ms
   [junit4]   2> 2859276 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=62519}
   [junit4]   2> 2859277 ERROR 
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging 
may be missing or incomplete.
   [junit4]   2> 2859277 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2859277 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? 
version 8.0.0
   [junit4]   2> 2859277 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on 
port null
   [junit4]   2> 2859278 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2859278 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-03-29T00:36:44.105Z
   [junit4]   2> 2859282 INFO  (zkConnectionManagerCallback-5376-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2859285 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2859318 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:62515/solr
   [junit4]   2> 2859319 INFO  (zkConnectionManagerCallback-5380-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2859323 INFO  (zkConnectionManagerCallback-5382-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2859335 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [n:127.0.0.1:62519_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (3)
   [junit4]   2> 2859339 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [n:127.0.0.1:62519_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2859343 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [n:127.0.0.1:62519_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 2147483647 transient cores
   [junit4]   2> 2859343 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [n:127.0.0.1:62519_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:62519_solr
   [junit4]   2> 2859345 INFO  (zkCallback-5343-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2859346 INFO  (zkCallback-5367-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2859346 INFO  (zkCallback-5355-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2859346 INFO  (zkCallback-5341-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2859346 INFO  (zkCallback-5363-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2859346 INFO  (zkCallback-5345-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2859346 INFO  (zkCallback-5373-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2859346 INFO  (zkCallback-5381-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2859446 INFO  (zkCallback-5343-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2859446 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2859639 INFO  
(OverseerCollectionConfigSetProcessor-72140217251528713-127.0.0.1:62518_solr-n_0000000000)
 [    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 2859700 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [n:127.0.0.1:62519_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62519.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2859725 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [n:127.0.0.1:62519_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62519.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2859725 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [n:127.0.0.1:62519_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_62519.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2859731 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [n:127.0.0.1:62519_solr    ] o.a.s.c.CorePropertiesLocator Found 1 core 
definitions underneath 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node2/.
   [junit4]   2> 2859731 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [n:127.0.0.1:62519_solr    ] o.a.s.c.CorePropertiesLocator Cores are: 
[raceDeleteReplica_shard1_replica_n2]
   [junit4]   2> 2859746 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 2859750 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr    ] o.a.s.s.IndexSchema 
[raceDeleteReplica_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 2859754 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with 
uniqueid field id
   [junit4]   2> 2859756 INFO  (zkConnectionManagerCallback-5387-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2859760 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [n:127.0.0.1:62519_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (4)
   [junit4]   2> 2859763 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene 
MatchVersion: 8.0.0
   [junit4]   2> 2859763 INFO  
(TEST-DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica-seed#[5B3BA86FA862E0EC])
 [n:127.0.0.1:62519_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:62515/solr ready
   [junit4]   2> 2859767 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.s.IndexSchema 
[raceDeleteReplica_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 2859769 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema 
minimal/1.1 with uniqueid field id
   [junit4]   2> 2859769 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 
'raceDeleteReplica_shard1_replica_n2' using configuration from collection 
raceDeleteReplica, trusted=true
   [junit4]   2> 2859769 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring 
for 'solr_62519.solr.core.raceDeleteReplica.shard1.replica_n2' (registry 
'solr.core.raceDeleteReplica.shard1.replica_n2') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2859769 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 2859769 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.SolrCore 
[[raceDeleteReplica_shard1_replica_n2] ] Opening new SolrCore at 
[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node2/raceDeleteReplica_shard1_replica_n2],
 
dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node2/./raceDeleteReplica_shard1_replica_n2/data/]
   [junit4]   2> 2859936 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2859936 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2859937 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 2859937 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 2859946 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@3d52ce0c[raceDeleteReplica_shard1_replica_n2] main]
   [junit4]   2> 2859949 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2859950 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2859950 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will 
be reserved for 10000ms.
   [junit4]   2> 2859951 INFO  
(coreLoadExecutor-9202-thread-1-processing-n:127.0.0.1:62519_solr) 
[n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1596230262878896128
   [junit4]   2> 2859951 INFO  
(searcherExecutor-9203-thread-1-processing-n:127.0.0.1:62519_solr 
x:raceDeleteReplica_shard1_replica_n2 c:raceDeleteReplica s:shard1 
r:core_node4) [n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.SolrCore 
[raceDeleteReplica_shard1_replica_n2] Registered new searcher 
Searcher@3d52ce0c[raceDeleteReplica_shard1_replica_n2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2859953 INFO  
(coreZkRegister-9197-thread-1-processing-n:127.0.0.1:62519_solr 
x:raceDeleteReplica_shard1_replica_n2 c:raceDeleteReplica s:shard1 
r:core_node4) [n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.DeleteReplicaTest Running delete 
core 
CoreDescriptor[name=raceDeleteReplica_shard1_replica_n2;instanceDir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node2/./raceDeleteReplica_shard1_replica_n2]
   [junit4]   2> 2860057 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2860057 INFO  (zkCallback-5343-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2860057 INFO  (zkCallback-5381-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2860057 INFO  (zkCallback-5381-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2860562 INFO  
(coreZkRegister-9197-thread-1-processing-n:127.0.0.1:62519_solr 
x:raceDeleteReplica_shard1_replica_n2 c:raceDeleteReplica s:shard1 
r:core_node4) [n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update 
of terms at /collections/raceDeleteReplica/terms/shard1 to 
Terms{values={core_node3=0}, version=2}
   [junit4]   2> 2860563 INFO  
(coreZkRegister-9197-thread-1-processing-n:127.0.0.1:62519_solr 
x:raceDeleteReplica_shard1_replica_n2 c:raceDeleteReplica s:shard1 
r:core_node4) [n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase 
No version found for ephemeral leader parent node, won't remove previous leader 
registration.
   [junit4]   2> 2860564 ERROR 
(coreZkRegister-9197-thread-1-processing-n:127.0.0.1:62519_solr 
x:raceDeleteReplica_shard1_replica_n2 c:raceDeleteReplica s:shard1 
r:core_node4) [n:127.0.0.1:62519_solr c:raceDeleteReplica s:shard1 r:core_node4 
x:raceDeleteReplica_shard1_replica_n2] o.a.s.c.ZkContainer 
:java.lang.NullPointerException
   [junit4]   2>        at 
org.apache.solr.cloud.ZkController.register(ZkController.java:1096)
   [junit4]   2>        at 
org.apache.solr.cloud.ZkController.register(ZkController.java:1019)
   [junit4]   2>        at 
org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:187)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 2860669 INFO  (zkCallback-5343-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2860669 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2860669 INFO  (zkCallback-5373-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2860671 INFO  (qtp1124586875-23567) [n:127.0.0.1:62521_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with 
params 
action=ADDREPLICA&collection=raceDeleteReplica&shard=shard1&type=NRT&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 2860677 INFO  (OverseerThreadFactory-9176-thread-2) [    ] 
o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:62521_solr for creating new 
replica
   [junit4]   2> 2860683 INFO  
(OverseerStateUpdate-72140217251528713-127.0.0.1:62518_solr-n_0000000000) [    
] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"addreplica",
   [junit4]   2>   "collection":"raceDeleteReplica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"raceDeleteReplica_shard1_replica_n5",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:62521/solr";,
   [junit4]   2>   "node_name":"127.0.0.1:62521_solr",
   [junit4]   2>   "type":"NRT"} 
   [junit4]   2> 2860794 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2860794 INFO  (zkCallback-5343-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2860907 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&name=raceDeleteReplica_shard1_replica_n5&action=CREATE&collection=raceDeleteReplica&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2860944 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.c.SolrConfig Using Lucene 
MatchVersion: 8.0.0
   [junit4]   2> 2860949 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.s.IndexSchema 
[raceDeleteReplica_shard1_replica_n5] Schema name=minimal
   [junit4]   2> 2860952 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.s.IndexSchema Loaded schema 
minimal/1.1 with uniqueid field id
   [junit4]   2> 2860952 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.c.CoreContainer Creating SolrCore 
'raceDeleteReplica_shard1_replica_n5' using configuration from collection 
raceDeleteReplica, trusted=true
   [junit4]   2> 2860952 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring 
for 'solr_62521.solr.core.raceDeleteReplica.shard1.replica_n5' (registry 
'solr.core.raceDeleteReplica.shard1.replica_n5') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@62f36ffe
   [junit4]   2> 2860952 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 2860952 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.c.SolrCore 
[[raceDeleteReplica_shard1_replica_n5] ] Opening new SolrCore at 
[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node4/raceDeleteReplica_shard1_replica_n5],
 
dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001/tempDir-001/node4/./raceDeleteReplica_shard1_replica_n5/data/]
   [junit4]   2> 2861015 INFO  (zkCallback-5345-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2861015 INFO  (zkCallback-5343-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2861015 INFO  (zkCallback-5343-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2861015 INFO  (zkCallback-5345-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/raceDeleteReplica/state.json] for collection 
[raceDeleteReplica] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2861027 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2861027 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2861044 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 2861044 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 2861045 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@49c8bd8f[raceDeleteReplica_shard1_replica_n5] main]
   [junit4]   2> 2861046 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2861046 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.r.ManagedResourceStorage Loaded 
null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2861047 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.h.ReplicationHandler Commits will 
be reserved for 10000ms.
   [junit4]   2> 2861047 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.u.UpdateLog Could not find max 
version in index or recent updates, using new clock 1596230264028135424
   [junit4]   2> 2861047 INFO  
(searcherExecutor-9208-thread-1-processing-n:127.0.0.1:62521_solr 
x:raceDeleteReplica_shard1_replica_n5 c:raceDeleteReplica s:shard1 
r:core_node6) [n:127.0.0.1:62521_solr c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.c.SolrCore 
[raceDeleteReplica_shard1_replica_n5] Registered new searcher 
Searcher@49c8bd8f[raceDeleteReplica_shard1_replica_n5] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2861051 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.c.ZkShardTerms Successful update 
of terms at /collections/raceDeleteReplica/terms/shard1 to 
Terms{values={core_node6=0, core_node3=0}, version=3}
   [junit4]   2> 2861054 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.c.ZkController Core needs to 
recover:raceDeleteReplica_shard1_replica_n5
   [junit4]   2> 2861054 INFO  
(updateExecutor-5335-thread-1-processing-n:127.0.0.1:62521_solr 
x:raceDeleteReplica_shard1_replica_n5 c:raceDeleteReplica s:shard1 
r:core_node6) [n:127.0.0.1:62521_solr c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.u.DefaultSolrCoreState Running 
recovery
   [junit4]   2> 2861055 INFO  
(recoveryExecutor-5338-thread-1-processing-n:127.0.0.1:62521_solr 
x:raceDeleteReplica_shard1_replica_n5 c:raceDeleteReplica s:shard1 
r:core_node6) [n:127.0.0.1:62521_solr c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.c.RecoveryStrategy Starting 
recovery process. recoveringAfterStartup=true
   [junit4]   2> 2861055 INFO  
(recoveryExecutor-5338-thread-1-processing-n:127.0.0.1:62521_solr 
x:raceDeleteReplica_shard1_replica_n5 c:raceDeleteReplica s:shard1 
r:core_node6) [n:127.0.0.1:62521_solr c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.c.RecoveryStrategy ###### 
startupVersions=[[]]
   [junit4]   2> 2861056 INFO  (qtp1124586875-23570) [n:127.0.0.1:62521_solr 
c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.s.HttpSolrCall [admin] webapp=null 
path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&name=raceDeleteReplica_shard1_replica_n5&action=CREATE&collection=raceDeleteReplica&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=149
   [junit4]   2> 2861060 INFO  (qtp1124586875-23567) [n:127.0.0.1:62521_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={action=ADDREPLICA&collection=raceDeleteReplica&shard=shard1&type=NRT&wt=javabin&version=2}
 status=0 QTime=389
   [junit4]   2> 2861061 INFO  (qtp559027215-23555) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.S.Request 
[raceDeleteReplica_shard1_replica_n1]  webapp=/solr path=/admin/ping 
params={wt=javabin&version=2} hits=0 status=0 QTime=1
   [junit4]   2> 2861061 INFO  (qtp559027215-23555) [n:127.0.0.1:62517_solr 
c:raceDeleteReplica s:shard1 r:core_node3 
x:raceDeleteReplica_shard1_replica_n1] o.a.s.c.S.Request 
[raceDeleteReplica_shard1_replica_n1]  webapp=/solr path=/admin/ping 
params={wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 2861062 INFO  
(recoveryExecutor-5338-thread-1-processing-n:127.0.0.1:62521_solr 
x:raceDeleteReplica_shard1_replica_n5 c:raceDeleteReplica s:shard1 
r:core_node6) [n:127.0.0.1:62521_solr c:raceDeleteReplica s:shard1 r:core_node6 
x:raceDeleteReplica_shard1_replica_n5] o.a.s.c.RecoveryStrategy Begin buffering 
updates. core=[raceDeleteReplica_shard1_replica

[...truncated too long message...]

r state changes
   [junit4]   2> 2922468 INFO  
(SUITE-DeleteReplicaTest-seed#[5B3BA86FA862E0EC]-worker) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:62515 62515
   [junit4]   2> 2922507 INFO  (Thread-4143) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:62515 62515
   [junit4]   2> 2922594 WARN  (Thread-4143) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        20      /solr/collections/raceDeleteReplica/terms/shard1
   [junit4]   2>        15      /solr/configs/conf
   [junit4]   2>        13      /solr/aliases.json
   [junit4]   2>        13      
/solr/collections/deleteReplicaOnIndexing/terms/shard1
   [junit4]   2>        13      /solr/clusterprops.json
   [junit4]   2>        7       /solr/collections/delLiveColl/terms/shard1
   [junit4]   2>        7       
/solr/collections/deletereplica_test/terms/shard1
   [junit4]   2>        6       /solr/collections/deleteByCountNew/terms/shard1
   [junit4]   2>        6       /solr/security.json
   [junit4]   2>        6       /solr/collections/deleteByCountNew/terms/shard2
   [junit4]   2>        5       /solr/collections/deleteByCount/terms/shard1
   [junit4]   2>        5       /solr/collections/delLiveColl/terms/shard2
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        39      /solr/collections/raceDeleteReplica/state.json
   [junit4]   2>        38      /solr/collections/delLiveColl/state.json
   [junit4]   2>        38      /solr/collections/deleteByCountNew/state.json
   [junit4]   2>        21      /solr/collections/deleteByCount/state.json
   [junit4]   2>        16      /solr/collections/deletereplica_test/state.json
   [junit4]   2>        14      
/solr/collections/deleteReplicaOnIndexing/state.json
   [junit4]   2>        13      /solr/clusterstate.json
   [junit4]   2>        5       
/solr/overseer_elect/election/72140217251528713-127.0.0.1:62518_solr-n_0000000000
   [junit4]   2>        3       
/solr/collections/raceDeleteReplica/leader_elect/shard1/election/72140217251528714-core_node3-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        13      /solr/live_nodes
   [junit4]   2>        13      /solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_5B3BA86FA862E0EC-001
   [junit4]   2> Mar 29, 2018 12:37:47 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
{id=PostingsFormat(name=Direct)}, docValues:{}, maxPointsInLeafNode=925, 
maxMBSortInHeap=5.458539401419171, 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@fafbff),
 locale=de-GR, timezone=Indian/Reunion
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_144 
(64-bit)/cpus=3,threads=1,free=168813136,total=515375104
   [junit4]   2> NOTE: All tests run in this JVM: [TestPhraseSuggestions, 
UniqFieldsUpdateProcessorFactoryTest, AddBlockUpdateTest, 
LeaderElectionIntegrationTest, TestLFUCache, SchemaVersionSpecificBehaviorTest, 
TestCoreBackup, HdfsRecoveryZkTest, CdcrUpdateLogTest, V2ApiIntegrationTest, 
LegacyCloudClusterPropTest, TestLegacyFieldCache, TestFieldCacheSort, 
SimpleCollectionCreateDeleteTest, QueryResultKeyTest, TestInitParams, 
TestRealTimeGet, OpenExchangeRatesOrgProviderTest, ActionThrottleTest, 
PolyFieldTest, DistributedSuggestComponentTest, 
DistributedFacetExistsSmallTest, ExitableDirectoryReaderTest, 
DistribJoinFromCollectionTest, TestIBSimilarityFactory, 
SolrMetricsIntegrationTest, TestSolrJ, TestExactStatsCache, 
WrapperMergePolicyFactoryTest, BlockJoinFacetRandomTest, 
CdcrVersionReplicationTest, SampleTest, HighlighterMaxOffsetTest, 
TestTolerantUpdateProcessorRandomCloud, TlogReplayBufferedWhileIndexingTest, 
SaslZkACLProviderTest, TestMiniSolrCloudClusterSSL, TestNamedUpdateProcessors, 
TestRecovery, TriggerIntegrationTest, SystemLogListenerTest, MetricTriggerTest, 
DeleteShardTest, TestReplicationHandler, TestSuggestSpellingConverter, 
CurrencyRangeFacetCloudTest, TestCursorMarkWithoutUniqueKey, 
BigEndianAscendingWordSerializerTest, ExplicitHLLTest, RAMDirectoryFactoryTest, 
DirectSolrSpellCheckerTest, OutOfBoxZkACLAndCredentialsProvidersTest, 
SolrRequestParserTest, RegexBytesRefFilterTest, TestRangeQuery, 
BasicDistributedZkTest, DistributedFacetPivotSmallTest, TestCloudSchemaless, 
NodeLostTriggerTest, SignatureUpdateProcessorFactoryTest, 
AsyncCallRequestStatusResponseTest, TestPullReplicaErrorHandling, 
TestXIncludeConfig, TestCSVResponseWriter, TestExecutePlanAction, 
SolrInfoBeanTest, RankQueryTest, BJQParserTest, 
TestSubQueryTransformerCrossCore, TestZkChroot, TestGroupingSearch, 
TestManagedSynonymFilterFactory, TestConfigSetProperties, 
ReplaceNodeNoTargetTest, ShardRoutingCustomTest, RuleEngineTest, 
TestNodeLostTrigger, TermsComponentTest, TestBulkSchemaConcurrent, 
TestBM25SimilarityFactory, TestEmbeddedSolrServerConstructors, 
CollectionsAPIAsyncDistributedZkTest, CdcrReplicationHandlerTest, 
TestSizeLimitedDistributedMap, TestImpersonationWithHadoopAuth, 
TestSmileRequest, AtomicUpdateProcessorFactoryTest, 
TestPKIAuthenticationPlugin, OpenCloseCoreStressTest, 
ParsingFieldUpdateProcessorsTest, ClassificationUpdateProcessorTest, 
TestTolerantSearch, SparseHLLTest, TestComputePlanAction, 
AlternateDirectoryTest, TestRequestStatusCollectionAPI, 
SpellingQueryConverterTest, TestCustomSort, TestTlogReplica, MetricsConfigTest, 
TestPostingsSolrHighlighter, TestCloudInspectUtil, 
HdfsChaosMonkeyNothingIsSafeTest, TestCharFilters, 
TestSlowCompositeReaderWrapper, OverseerTaskQueueTest, TestSolrDeletionPolicy1, 
SolrIndexSplitterTest, DeleteInactiveReplicaTest, ForceLeaderTest, 
VMParamsZkACLAndCredentialsProvidersTest, TestSolrCloudWithSecureImpersonation, 
ConfigSetsAPITest, ClassificationUpdateProcessorFactoryTest, 
BasicAuthIntegrationTest, TestFilteredDocIdSet, SuggesterFSTTest, 
TestJsonFacetRefinement, TestLeaderInitiatedRecoveryThread, 
OverseerCollectionConfigSetProcessorTest, EnumFieldTest, 
StatelessScriptUpdateProcessorFactoryTest, FastVectorHighlighterTest, 
DateMathParserTest, DistributedQueryComponentCustomSortTest, TestPrepRecovery, 
ConcurrentDeleteAndCreateCollectionTest, TestCollationField, 
BinaryUpdateRequestHandlerTest, DistributedMLTComponentTest, DeleteNodeTest, 
TestFieldCache, DistanceUnitsTest, CollectionReloadTest, 
TestPayloadCheckQParserPlugin, TestNoOpRegenerator, TestReload, 
CoreMergeIndexesAdminHandlerTest, DirectUpdateHandlerOptimizeTest, 
TestSimpleTextCodec, BufferStoreTest, SecurityConfHandlerTest, TestJoin, 
TestJmxIntegration, DocumentBuilderTest, TestRestoreCore, RecoveryZkTest, 
PingRequestHandlerTest, QueryElevationComponentTest, 
TestLocalFSCloudBackupRestore, TestHdfsUpdateLog, AliasIntegrationTest, 
SolrCloudReportersTest, MBeansHandlerTest, UnloadDistributedZkTest, 
MissingSegmentRecoveryTest, RollingRestartTest, TestDistribIDF, TestFiltering, 
AnalyticsMergeStrategyTest, TestCloudRecovery, CursorPagingTest, 
TestRestManager, TestSweetSpotSimilarityFactory, HdfsNNFailoverTest, 
TestMergePolicyConfig, PeerSyncWithIndexFingerprintCachingTest, 
SolrXmlInZkTest, TestHashQParserPlugin, TestCollectionsAPIViaSolrCloudCluster, 
TestLazyCores, TestXmlQParserPlugin, ZkShardTermsTest, 
LeaderFailureAfterFreshStartTest, TestLegacyNumericRangeQueryBuilder, 
HdfsCollectionsAPIDistributedZkTest, ChangedSchemaMergeTest, TestFieldResource, 
IndexSchemaRuntimeFieldTest, TestElisionMultitermQuery, OutputWriterTest, 
TestLargeCluster, TestDocBasedVersionConstraints, TestHalfAndHalfDocValues, 
ZkCLITest, TestCloudConsistency, TestSchemalessBufferedUpdates, 
TestMissingGroups, PKIAuthenticationIntegrationTest, SolrGraphiteReporterTest, 
SpellCheckCollatorWithCollapseTest, PathHierarchyTokenizerFactoryTest, 
TestSolrDeletionPolicy2, MultiTermTest, TestDocSet, TestBinaryField, 
TestSearchPerf, NumericFieldsTest, TestConfig, TestFuzzyAnalyzedSuggestions, 
DirectSolrConnectionTest, SpellPossibilityIteratorTest, TestCodecSupport, 
SynonymTokenizerTest, TestDFRSimilarityFactory, TestPerFieldSimilarity, 
TestLMJelinekMercerSimilarityFactory, TestFastWriter, TimeZoneUtilsTest, 
ResourceLoaderTest, TestFastOutputStream, URLClassifyProcessorTest, 
PluginInfoTest, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, PrimUtilsTest, 
DateFieldTest, TestDocumentBuilder, FileUtilsTest, TestHighlightDedupGrouping, 
TokenizerChainTest, TestLuceneIndexBackCompat, TestJettySolrRunner, 
AssignBackwardCompatibilityTest, ChaosMonkeySafeLeaderWithPullReplicasTest, 
CleanupOldIndexTest, CollectionPropsTest, ConcurrentCreateRoutedAliasTest, 
CreateCollectionCleanupTest, CreateRoutedAliasTest, DeleteReplicaTest]
   [junit4] Completed [530/786 (1!)] on J0 in 69.00s, 6 tests, 1 error <<< 
FAILURES!

[...truncated 50426 lines...]
[repro] Jenkins log URL: 
https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4533/consoleText

[repro] Revision: 060d82af316bd2512e3f014365ce82db6c2f9fdf

[repro] Ant options: "-Dargs=-XX:+UseCompressedOops -XX:+UseParallelGC"
[repro] Repro line:  ant test  -Dtestcase=DeleteReplicaTest 
-Dtests.method=deleteReplicaOnIndexing -Dtests.seed=5B3BA86FA862E0EC 
-Dtests.slow=true -Dtests.locale=de-GR -Dtests.timezone=Indian/Reunion 
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       DeleteReplicaTest
[repro] ant compile-test

[...truncated 3317 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.DeleteReplicaTest" -Dtests.showOutput=onerror 
"-Dargs=-XX:+UseCompressedOops -XX:+UseParallelGC" 
-Dtests.seed=5B3BA86FA862E0EC -Dtests.slow=true -Dtests.locale=de-GR 
-Dtests.timezone=Indian/Reunion -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[...truncated 93 lines...]
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.DeleteReplicaTest
[repro] Exiting with code 0

[...truncated 70 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to