Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/520/

1 tests failed.
FAILED:  org.apache.solr.cloud.TestPullReplica.testKillLeader

Error Message:
Replica core_node4 not up to date after 10 seconds expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: Replica core_node4 not up to date after 10 seconds 
expected:<1> but was:<0>
        at 
__randomizedtesting.SeedInfo.seed([EEB123C12B731A74:A7A7D77549C88E22]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at 
org.apache.solr.cloud.TestPullReplica.waitForNumDocsInAllReplicas(TestPullReplica.java:538)
        at 
org.apache.solr.cloud.TestPullReplica.doTestNoLeader(TestPullReplica.java:486)
        at 
org.apache.solr.cloud.TestPullReplica.testKillLeader(TestPullReplica.java:305)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1787 lines...]
   [junit4] JVM J0: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J0-20180320_162145_197600697336804868996.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] codec: Lucene70, pf: Asserting, dvf: Memory
   [junit4] <<< JVM J0: EOF ----

[...truncated 12273 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestPullReplica
   [junit4]   2> 1829208 INFO  
(SUITE-TestPullReplica-seed#[EEB123C12B731A74]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_EEB123C12B731A74-001/init-core-data-001
   [junit4]   2> 1829208 WARN  
(SUITE-TestPullReplica-seed#[EEB123C12B731A74]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 1829209 INFO  
(SUITE-TestPullReplica-seed#[EEB123C12B731A74]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1829210 INFO  
(SUITE-TestPullReplica-seed#[EEB123C12B731A74]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1829211 INFO  
(SUITE-TestPullReplica-seed#[EEB123C12B731A74]-worker) [    ] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_EEB123C12B731A74-001/tempDir-001
   [junit4]   2> 1829211 INFO  
(SUITE-TestPullReplica-seed#[EEB123C12B731A74]-worker) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1829211 INFO  (Thread-4701) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1829211 INFO  (Thread-4701) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 1829214 ERROR (Thread-4701) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1829311 INFO  
(SUITE-TestPullReplica-seed#[EEB123C12B731A74]-worker) [    ] 
o.a.s.c.ZkTestServer start zk server on port:45143
   [junit4]   2> 1829314 INFO  (zkConnectionManagerCallback-4058-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1829319 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1829319 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1829320 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1829320 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1829320 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1829322 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@43d77d22{/solr,null,AVAILABLE}
   [junit4]   2> 1829322 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1829322 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1829322 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1829322 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@4f657b84{HTTP/1.1,[http/1.1]}{127.0.0.1:45921}
   [junit4]   2> 1829322 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.e.j.s.Server Started @1836199ms
   [junit4]   2> 1829323 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=45921}
   [junit4]   2> 1829323 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@1f2d0e96{/solr,null,AVAILABLE}
   [junit4]   2> 1829323 ERROR (jetty-launcher-4055-thread-2) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1829323 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.4.0
   [junit4]   2> 1829323 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@53363126{HTTP/1.1,[http/1.1]}{127.0.0.1:34355}
   [junit4]   2> 1829323 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1829323 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.e.j.s.Server Started @1836200ms
   [junit4]   2> 1829323 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1829323 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=34355}
   [junit4]   2> 1829323 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-03-20T17:24:42.032Z
   [junit4]   2> 1829323 ERROR (jetty-launcher-4055-thread-1) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1829323 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.4.0
   [junit4]   2> 1829324 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1829324 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1829324 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-03-20T17:24:42.033Z
   [junit4]   2> 1829325 INFO  (zkConnectionManagerCallback-4062-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1829326 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1829331 INFO  (zkConnectionManagerCallback-4060-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1829332 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1829335 INFO  (jetty-launcher-4055-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45143/solr
   [junit4]   2> 1829336 INFO  (zkConnectionManagerCallback-4066-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1829339 INFO  
(zkConnectionManagerCallback-4068-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1829343 INFO  (jetty-launcher-4055-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45143/solr
   [junit4]   2> 1829345 INFO  (zkConnectionManagerCallback-4072-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1829349 INFO  
(zkConnectionManagerCallback-4074-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1829466 INFO  (jetty-launcher-4055-thread-1) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1829466 INFO  (jetty-launcher-4055-thread-2) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1829467 INFO  (jetty-launcher-4055-thread-1) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:34355_solr
   [junit4]   2> 1829468 INFO  (jetty-launcher-4055-thread-2) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:45921_solr
   [junit4]   2> 1829468 INFO  (jetty-launcher-4055-thread-1) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.Overseer Overseer 
(id=73487496792965124-127.0.0.1:34355_solr-n_0000000000) starting
   [junit4]   2> 1829474 INFO  
(zkCallback-4067-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1829474 INFO  
(zkCallback-4073-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1829481 INFO  
(zkConnectionManagerCallback-4080-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1829482 INFO  (jetty-launcher-4055-thread-1) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1829483 INFO  (jetty-launcher-4055-thread-1) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:45143/solr ready
   [junit4]   2> 1829495 INFO  (jetty-launcher-4055-thread-1) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:34355_solr
   [junit4]   2> 1829496 INFO  
(zkCallback-4073-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1829497 INFO  
(zkCallback-4067-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1829501 INFO  
(zkCallback-4079-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1829950 INFO  (jetty-launcher-4055-thread-1) 
[n:127.0.0.1:34355_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_34355.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@33d00b90
   [junit4]   2> 1829963 INFO  (jetty-launcher-4055-thread-1) 
[n:127.0.0.1:34355_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_34355.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@33d00b90
   [junit4]   2> 1829963 INFO  (jetty-launcher-4055-thread-1) 
[n:127.0.0.1:34355_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_34355.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@33d00b90
   [junit4]   2> 1829964 INFO  (jetty-launcher-4055-thread-1) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_EEB123C12B731A74-001/tempDir-001/node1/.
   [junit4]   2> 1830569 INFO  (jetty-launcher-4055-thread-2) 
[n:127.0.0.1:45921_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_45921.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@33d00b90
   [junit4]   2> 1830581 INFO  (jetty-launcher-4055-thread-2) 
[n:127.0.0.1:45921_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_45921.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@33d00b90
   [junit4]   2> 1830581 INFO  (jetty-launcher-4055-thread-2) 
[n:127.0.0.1:45921_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_45921.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@33d00b90
   [junit4]   2> 1830583 INFO  (jetty-launcher-4055-thread-2) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_EEB123C12B731A74-001/tempDir-001/node2/.
   [junit4]   2> 1830586 INFO  
(zkConnectionManagerCallback-4084-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1830587 INFO  (jetty-launcher-4055-thread-2) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 1830588 INFO  (jetty-launcher-4055-thread-2) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:45143/solr ready
   [junit4]   2> 1830619 INFO  (zkConnectionManagerCallback-4086-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1830622 INFO  (zkConnectionManagerCallback-4090-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1830623 INFO  
(SUITE-TestPullReplica-seed#[EEB123C12B731A74]-worker) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1830624 INFO  
(SUITE-TestPullReplica-seed#[EEB123C12B731A74]-worker) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45143/solr ready
   [junit4]   2> 1830628 INFO  
(SUITE-TestPullReplica-seed#[EEB123C12B731A74]-worker) [    ] 
o.a.s.c.TestPullReplica Using legacyCloud?: false
   [junit4]   2> 1830629 INFO  (qtp513656487-17205) [n:127.0.0.1:45921_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with 
params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and 
sendToOCPQueue=true
   [junit4]   2> 1830630 INFO  (qtp513656487-17205) [n:127.0.0.1:45921_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} 
status=0 QTime=0
   [junit4]   2> 1830662 INFO  
(TEST-TestPullReplica.testSearchWhileReplicationHappens-seed#[EEB123C12B731A74])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testSearchWhileReplicationHappens
   [junit4]   2> 1830663 INFO  
(TEST-TestPullReplica.testSearchWhileReplicationHappens-seed#[EEB123C12B731A74])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testSearchWhileReplicationHappens
   [junit4]   2> 1830689 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testKillPullReplica
   [junit4]   2> 1830690 INFO  (qtp513656487-17208) [n:127.0.0.1:45921_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
pullReplicas=1&replicationFactor=1&collection.configName=conf&maxShardsPerNode=100&name=pull_replica_test_kill_pull_replica&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1830692 INFO  
(OverseerThreadFactory-5762-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection 
pull_replica_test_kill_pull_replica
   [junit4]   2> 1830806 INFO  
(OverseerStateUpdate-73487496792965124-127.0.0.1:34355_solr-n_0000000000) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"pull_replica_test_kill_pull_replica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   
"core":"pull_replica_test_kill_pull_replica_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:34355/solr";,
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 1830809 INFO  
(OverseerStateUpdate-73487496792965124-127.0.0.1:34355_solr-n_0000000000) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"pull_replica_test_kill_pull_replica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   
"core":"pull_replica_test_kill_pull_replica_shard1_replica_p2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:45921/solr";,
   [junit4]   2>   "type":"PULL",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 1831012 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=pull_replica_test_kill_pull_replica_shard1_replica_p2&action=CREATE&numShards=1&collection=pull_replica_test_kill_pull_replica&shard=shard1&wt=javabin&version=2&replicaType=PULL
   [junit4]   2> 1831012 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 1831013 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=pull_replica_test_kill_pull_replica_shard1_replica_n1&action=CREATE&numShards=1&collection=pull_replica_test_kill_pull_replica&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1831013 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 1831121 INFO  
(zkCallback-4073-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1831121 INFO  
(zkCallback-4067-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1831121 INFO  
(zkCallback-4073-thread-2-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1831121 INFO  
(zkCallback-4067-thread-2-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1832031 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 1832034 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 1832042 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.s.IndexSchema 
[pull_replica_test_kill_pull_replica_shard1_replica_p2] Schema name=minimal
   [junit4]   2> 1832045 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.s.IndexSchema 
Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1832045 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.CoreContainer 
Creating SolrCore 'pull_replica_test_kill_pull_replica_shard1_replica_p2' using 
configuration from collection pull_replica_test_kill_pull_replica, trusted=true
   [junit4]   2> 1832046 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_45921.solr.core.pull_replica_test_kill_pull_replica.shard1.replica_p2' 
(registry 'solr.core.pull_replica_test_kill_pull_replica.shard1.replica_p2') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@33d00b90
   [junit4]   2> 1832046 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 1832046 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.SolrCore 
[[pull_replica_test_kill_pull_replica_shard1_replica_p2] ] Opening new SolrCore 
at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_EEB123C12B731A74-001/tempDir-001/node2/pull_replica_test_kill_pull_replica_shard1_replica_p2],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_EEB123C12B731A74-001/tempDir-001/node2/./pull_replica_test_kill_pull_replica_shard1_replica_p2/data/]
   [junit4]   2> 1832049 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.s.IndexSchema 
[pull_replica_test_kill_pull_replica_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 1832051 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.s.IndexSchema 
Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1832051 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.CoreContainer 
Creating SolrCore 'pull_replica_test_kill_pull_replica_shard1_replica_n1' using 
configuration from collection pull_replica_test_kill_pull_replica, trusted=true
   [junit4]   2> 1832052 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_34355.solr.core.pull_replica_test_kill_pull_replica.shard1.replica_n1' 
(registry 'solr.core.pull_replica_test_kill_pull_replica.shard1.replica_n1') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@33d00b90
   [junit4]   2> 1832052 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 1832052 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SolrCore 
[[pull_replica_test_kill_pull_replica_shard1_replica_n1] ] Opening new SolrCore 
at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_EEB123C12B731A74-001/tempDir-001/node1/pull_replica_test_kill_pull_replica_shard1_replica_n1],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_EEB123C12B731A74-001/tempDir-001/node1/./pull_replica_test_kill_pull_replica_shard1_replica_n1/data/]
   [junit4]   2> 1832116 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 1832116 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1832116 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 1832116 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1832118 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 1832118 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 1832118 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@5d7e2860[pull_replica_test_kill_pull_replica_shard1_replica_p2] main]
   [junit4]   2> 1832120 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@64953e5c[pull_replica_test_kill_pull_replica_shard1_replica_n1] main]
   [junit4]   2> 1832120 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 1832120 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1832121 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 1832121 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1832121 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1832122 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1832123 INFO  
(searcherExecutor-5767-thread-1-processing-n:127.0.0.1:45921_solr 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2 s:shard1 
c:pull_replica_test_kill_pull_replica r:core_node4) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.SolrCore 
[pull_replica_test_kill_pull_replica_shard1_replica_p2] Registered new searcher 
Searcher@5d7e2860[pull_replica_test_kill_pull_replica_shard1_replica_p2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1832123 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1595478308114399232
   [junit4]   2> 1832123 INFO  
(searcherExecutor-5768-thread-1-processing-n:127.0.0.1:34355_solr 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1 s:shard1 
c:pull_replica_test_kill_pull_replica r:core_node3) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SolrCore 
[pull_replica_test_kill_pull_replica_shard1_replica_n1] Registered new searcher 
Searcher@64953e5c[pull_replica_test_kill_pull_replica_shard1_replica_n1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1832131 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.ZkController 
pull_replica_test_kill_pull_replica_shard1_replica_p2 starting background 
replication from leader
   [junit4]   2> 1832132 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.c.ReplicateFromLeader Will start replication from leader with poll 
interval: 00:00:03
   [junit4]   2> 1832132 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.ZkShardTerms 
Successful update of terms at 
/collections/pull_replica_test_kill_pull_replica/terms/shard1 to 
Terms{values={core_node3=0}, version=0}
   [junit4]   2> 1832135 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
   [junit4]   2> 1832135 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1832138 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1832138 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1832138 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SyncStrategy 
Sync replicas to 
http://127.0.0.1:34355/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/
   [junit4]   2> 1832138 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SyncStrategy 
Sync Success - now sync replicas to me
   [junit4]   2> 1832138 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SyncStrategy 
http://127.0.0.1:34355/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/
 has no replicas
   [junit4]   2> 1832138 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 1832143 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:34355/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/
 shard1
   [junit4]   2> 1832245 INFO  
(zkCallback-4073-thread-2-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1832245 INFO  
(zkCallback-4073-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1832246 INFO  
(zkCallback-4067-thread-2-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1832246 INFO  
(zkCallback-4067-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1832285 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher Replica core_node3 is leader but it's state is down, 
skipping replication
   [junit4]   2> 1832296 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.ZkController I 
am the leader, no recovery necessary
   [junit4]   2> 1832298 INFO  (qtp1724234586-17206) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=pull_replica_test_kill_pull_replica_shard1_replica_n1&action=CREATE&numShards=1&collection=pull_replica_test_kill_pull_replica&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=1285
   [junit4]   2> 1832399 INFO  
(zkCallback-4073-thread-2-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1832399 INFO  
(zkCallback-4067-thread-2-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1832399 INFO  
(zkCallback-4067-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1832399 INFO  
(zkCallback-4073-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1833139 INFO  (qtp513656487-17212) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=pull_replica_test_kill_pull_replica_shard1_replica_p2&action=CREATE&numShards=1&collection=pull_replica_test_kill_pull_replica&shard=shard1&wt=javabin&version=2&replicaType=PULL}
 status=0 QTime=2127
   [junit4]   2> 1833142 INFO  (qtp513656487-17208) [n:127.0.0.1:45921_solr    
] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 1833241 INFO  
(zkCallback-4073-thread-2-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1833251 INFO  
(zkCallback-4073-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1833252 INFO  
(zkCallback-4067-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1833252 INFO  
(zkCallback-4067-thread-2-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1834143 INFO  (qtp513656487-17208) [n:127.0.0.1:45921_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={pullReplicas=1&replicationFactor=1&collection.configName=conf&maxShardsPerNode=100&name=pull_replica_test_kill_pull_replica&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2}
 status=0 QTime=3453
   [junit4]   2> 1834149 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1834151 INFO  (qtp513656487-17208) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1834155 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.ZkShardTerms 
Successful update of terms at 
/collections/pull_replica_test_kill_pull_replica/terms/shard1 to 
Terms{values={core_node3=1}, version=1}
   [junit4]   2> 1834156 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/update params={wt=javabin&version=2}{add=[1 (1595478310243008512)]} 0 2
   [junit4]   2> 1834156 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.u.DirectUpdateHandler2 start 
commit{_version_=1595478310246154240,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1834156 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.u.SolrIndexWriter Calling setCommitData with 
IW:org.apache.solr.update.SolrIndexWriter@2df8985d 
commitCommandVersion:1595478310246154240
   [junit4]   2> 1834163 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@77cc8f49[pull_replica_test_kill_pull_replica_shard1_replica_n1] main]
   [junit4]   2> 1834164 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1834165 INFO  
(searcherExecutor-5768-thread-1-processing-n:127.0.0.1:34355_solr 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1 s:shard1 
c:pull_replica_test_kill_pull_replica r:core_node3) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SolrCore 
[pull_replica_test_kill_pull_replica_shard1_replica_n1] Registered new searcher 
Searcher@77cc8f49[pull_replica_test_kill_pull_replica_shard1_replica_n1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.4.0):C1)))}
   [junit4]   2> 1834166 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/update 
params={_stateVer_=pull_replica_test_kill_pull_replica:5&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=}
 0 9
   [junit4]   2> 1834168 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
   [junit4]   2> 1834170 INFO  (qtp513656487-17208) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1834271 INFO  (qtp513656487-17214) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1834372 INFO  (qtp513656487-17208) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1834473 INFO  (qtp513656487-17214) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1834574 INFO  (qtp513656487-17208) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1834675 INFO  (qtp513656487-17214) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1834695 INFO  
(OverseerCollectionConfigSetProcessor-73487496792965124-127.0.0.1:34355_solr-n_0000000000)
 [n:127.0.0.1:34355_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 1834776 INFO  (qtp513656487-17208) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1834877 INFO  (qtp513656487-17214) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1834978 INFO  (qtp513656487-17208) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1835079 INFO  (qtp513656487-17214) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1835135 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
   [junit4]   2> 1835135 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher Updated masterUrl to 
http://127.0.0.1:34355/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/
   [junit4]   2> 1835136 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 
QTime=0
   [junit4]   2> 1835136 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher Master's generation: 2
   [junit4]   2> 1835136 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher Master's version: 1521566686865
   [junit4]   2> 1835136 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1835136 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 1835136 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1835137 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&tlogFiles=false&wt=javabin&version=2&command=filelist}
 status=0 QTime=0
   [junit4]   2> 1835137 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher Number of files in latest index in master: 20
   [junit4]   2> 1835138 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher Starting download (fullCopy=true) to 
MockDirectoryWrapper(RAMDirectory@2c7e0f10 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@395667c6)
   [junit4]   2> 1835138 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835140 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_Lucene50_0.doc&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835141 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_Lucene50_0.tim&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835142 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_Lucene50_0.pos&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835143 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835144 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835145 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_Lucene50_0.tip&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835145 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835146 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_Lucene70_0.dvm&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835147 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.dii&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835148 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_MockRandom_0.pos&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835149 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_MockRandom_0.doc&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835150 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_Lucene70_0.dvd&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835151 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_MockRandom_0.sd&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835152 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_MockRandom_0.tio&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835153 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.dim&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835154 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_MockRandom_0.tipo&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835154 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835155 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835156 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 1835156 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher Bytes downloaded: 2960, Bytes skipped downloading: 0
   [junit4]   2> 1835157 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher Total time taken for download 
(fullCopy=true,bytesDownloaded=2960) : 0 secs (null bytes/sec) to 
MockDirectoryWrapper(RAMDirectory@2c7e0f10 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@395667c6)
   [junit4]   2> 1835157 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.c.SolrCore Updating index properties... index=index.20180320182447847
   [junit4]   2> 1835158 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.IndexFetcher removing old index directory 
MockDirectoryWrapper(RAMDirectory@541a655b 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5c076ed4)
   [junit4]   2> 1835159 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1835160 INFO  (indexFetcher-5777-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@6a026a81[pull_replica_test_kill_pull_replica_shard1_replica_p2] main]
   [junit4]   2> 1835162 INFO  
(searcherExecutor-5767-thread-1-processing-n:127.0.0.1:45921_solr 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2 s:shard1 
c:pull_replica_test_kill_pull_replica r:core_node4) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.SolrCore 
[pull_replica_test_kill_pull_replica_shard1_replica_p2] Registered new searcher 
Searcher@6a026a81[pull_replica_test_kill_pull_replica_shard1_replica_p2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.4.0):C1)))}
   [junit4]   2> 1835180 INFO  (qtp513656487-17208) [n:127.0.0.1:45921_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 
x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
   [junit4]   2> 1835181 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.c.ChaosMonkey monkey: kill jetty! 45921
   [junit4]   2> 1835182 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.e.j.s.AbstractConnector Stopped 
ServerConnector@4f657b84{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1835182 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=752556894
   [junit4]   2> 1835183 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 1835183 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@1771213a: rootName = 
solr_45921, domain = solr.node, service url = null, agent id = null] for 
registry solr.node / com.codahale.metrics.MetricRegistry@5a3703e1
   [junit4]   2> 1835192 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 1835192 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@2b179b92: rootName = 
solr_45921, domain = solr.jvm, service url = null, agent id = null] for 
registry solr.jvm / com.codahale.metrics.MetricRegistry@4794f4b0
   [junit4]   2> 1835198 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 1835198 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@67b42560: rootName = 
solr_45921, domain = solr.jetty, service url = null, agent id = null] for 
registry solr.jetty / com.codahale.metrics.MetricRegistry@41d7d00d
   [junit4]   2> 1835200 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.c.ZkController Remove node as live in 
ZooKeeper:/live_nodes/127.0.0.1:45921_solr
   [junit4]   2> 1835201 INFO  
(zkCallback-4083-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (1)
   [junit4]   2> 1835202 INFO  
(zkCallback-4079-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (1)
   [junit4]   2> 1835202 INFO  (zkCallback-4089-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1835202 INFO  
(zkCallback-4067-thread-2-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (1)
   [junit4]   2> 1835202 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, 
tag=null
   [junit4]   2> 1835202 INFO  
(zkCallback-4073-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (1)
   [junit4]   2> 1835204 INFO  (coreCloseExecutor-5779-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.c.SolrCore [pull_replica_test_kill_pull_replica_shard1_replica_p2]  
CLOSING SolrCore org.apache.solr.core.SolrCore@7e856e41
   [junit4]   2> 1835204 INFO  (coreCloseExecutor-5779-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.core.pull_replica_test_kill_pull_replica.shard1.replica_p2, 
tag=2122673729
   [junit4]   2> 1835204 INFO  (coreCloseExecutor-5779-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@66d22fab: rootName = 
solr_45921, domain = 
solr.core.pull_replica_test_kill_pull_replica.shard1.replica_p2, service url = 
null, agent id = null] for registry 
solr.core.pull_replica_test_kill_pull_replica.shard1.replica_p2 / 
com.codahale.metrics.MetricRegistry@b482784
   [junit4]   2> 1835215 INFO  (coreCloseExecutor-5779-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.collection.pull_replica_test_kill_pull_replica.shard1.leader, 
tag=2122673729
   [junit4]   2> 1835218 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@43d77d22{/solr,null,UNAVAILABLE}
   [junit4]   2> 1835219 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.e.j.s.session Stopped scavenging
   [junit4]   2> 1835220 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.c.ChaosMonkey monkey: stop jetty! 45921
   [junit4]   2> 1835302 INFO  (zkCallback-4089-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [1])
   [junit4]   2> 1835302 INFO  
(zkCallback-4067-thread-2-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [1])
   [junit4]   2> 1835302 INFO  
(zkCallback-4067-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [1])
   [junit4]   2> 1835302 INFO  (zkCallback-4089-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [1])
   [junit4]   2> 1835305 INFO  (qtp1724234586-17217) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/update params={wt=javabin&version=2}{add=[2 (1595478311448870912)]} 0 2
   [junit4]   2> 1835306 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.u.DirectUpdateHandler2 start 
commit{_version_=1595478311452016640,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1835306 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.u.SolrIndexWriter Calling setCommitData with 
IW:org.apache.solr.update.SolrIndexWriter@2df8985d 
commitCommandVersion:1595478311452016640
   [junit4]   2> 1835309 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@307aeef9[pull_replica_test_kill_pull_replica_shard1_replica_n1] main]
   [junit4]   2> 1835309 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1835311 INFO  
(searcherExecutor-5768-thread-1-processing-n:127.0.0.1:34355_solr 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1 s:shard1 
c:pull_replica_test_kill_pull_replica r:core_node3) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.SolrCore 
[pull_replica_test_kill_pull_replica_shard1_replica_n1] Registered new searcher 
Searcher@307aeef9[pull_replica_test_kill_pull_replica_shard1_replica_n1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.4.0):C1)
 Uninverting(_1(7.4.0):C1)))}
   [junit4]   2> 1835311 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/update 
params={_stateVer_=pull_replica_test_kill_pull_replica:5&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=}
 0 5
   [junit4]   2> 1835314 INFO  (qtp1724234586-17219) [n:127.0.0.1:34355_solr 
c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 
x:pull_replica_test_kill_pull_replica_shard1_replica_n1] o.a.s.c.S.Request 
[pull_replica_test_kill_pull_replica_shard1_replica_n1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=2 status=0 QTime=0
   [junit4]   2> 1835315 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.c.ChaosMonkey monkey: starting jetty! 45921
   [junit4]   2> 1835315 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 
2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1835316 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1835316 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1835316 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1835316 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@21cf51aa{/solr,null,AVAILABLE}
   [junit4]   2> 1835317 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@337a5b2a{HTTP/1.1,[http/1.1]}{127.0.0.1:45921}
   [junit4]   2> 1835317 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.e.j.s.Server Started @1842194ms
   [junit4]   2> 1835317 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=45921}
   [junit4]   2> 1835317 ERROR 
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1835317 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.4.0
   [junit4]   2> 1835318 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1835318 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1835318 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-03-20T17:24:48.027Z
   [junit4]   2> 1835319 INFO  (zkConnectionManagerCallback-4092-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1835320 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1835329 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45143/solr
   [junit4]   2> 1835331 INFO  (zkConnectionManagerCallback-4096-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1835334 INFO  
(zkConnectionManagerCallback-4098-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1835341 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1835343 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1835346 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:45921_solr
   [junit4]   2> 1835351 INFO  (zkCallback-4089-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1835351 INFO  
(zkCallback-4079-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1835351 INFO  
(zkCallback-4067-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1835352 INFO  
(zkCallback-4097-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1835446 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) 
[n:127.0.0.1:45921_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_45921.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@33d00b90
   [junit4]   2> 1835448 INFO  
(zkCallback-4067-thread-1-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1835448 INFO  
(zkCallback-4067-thread-2-processing-n:127.0.0.1:34355_solr) 
[n:127.0.0.1:34355_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/pull_replica_test_kill_pull_replica/state.json] for 
collection [pull_replica_test_kill_pull_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 1835462 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) 
[n:127.0.0.1:45921_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_45921.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@33d00b90
   [junit4]   2> 1835463 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) 
[n:127.0.0.1:45921_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_45921.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@33d00b90
   [junit4]   2> 1835465 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.CorePropertiesLocator Found 1 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_EEB123C12B731A74-001/tempDir-001/node2/.
   [junit4]   2> 1835465 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.CorePropertiesLocator Cores are: 
[pull_replica_test_kill_pull_replica_shard1_replica_p2]
   [junit4]   2> 1835468 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 2147483647 transient cores
   [junit4]   2> 1835469 INFO  
(zkConnectionManagerCallback-4103-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1835471 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 1835472 INFO  
(TEST-TestPullReplica.testKillPullReplica-seed#[EEB123C12B731A74]) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:45143/solr ready
   [junit4]   2> 1835512 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 1835521 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.s.IndexSchema 
[pull_replica_test_kill_pull_replica_shard1_replica_p2] Schema name=minimal
   [junit4]   2> 1835523 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with 
uniqueid field id
   [junit4]   2> 1835554 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 1835564 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.s.IndexSchema [pull_replica_test_kill_pull_replica_shard1_replica_p2] 
Schema name=minimal
   [junit4]   2> 1835566 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1835567 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.c.CoreContainer Creating SolrCore 
'pull_replica_test_kill_pull_replica_shard1_replica_p2' using configuration 
from collection pull_replica_test_kill_pull_replica, trusted=true
   [junit4]   2> 1835567 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_45921.solr.core.pull_replica_test_kill_pull_replica.shard1.replica_p2' 
(registry 'solr.core.pull_replica_test_kill_pull_replica.shard1.replica_p2') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@33d00b90
   [junit4]   2> 1835567 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1835567 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.c.SolrCore [[pull_replica_test_kill_pull_replica_shard1_replica_p2] ] 
Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_EEB123C12B731A74-001/tempDir-001/node2/pull_replica_test_kill_pull_replica_shard1_replica_p2],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_EEB123C12B731A74-001/tempDir-001/node2/./pull_replica_test_kill_pull_replica_shard1_replica_p2/data/]
   [junit4]   2> 1835620 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1835620 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1835622 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@446923d0[pull_replica_test_kill_pull_replica_shard1_replica_p2] main]
   [junit4]   2> 1835623 INFO  
(coreLoadExecutor-5786-thread-1-processing-n:127.0.0.1:45921_solr) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/con

[...truncated too long message...]

.IndexFetcher.fetchLatestIndex(IndexFetcher.java:347)
   [junit4]   2>        at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
   [junit4]   2>        at 
org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$12(ReplicationHandler.java:1159)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1934602 ERROR (indexFetcher-5796-thread-1) 
[n:127.0.0.1:45921_solr c:pull_replica_test_kill_pull_replica s:shard1 
r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] 
o.a.s.h.ReplicationHandler Index fetch failed 
:org.apache.solr.common.SolrException: No registered leader was found after 
waiting for 4000ms , collection: pull_replica_test_kill_pull_replica slice: 
shard1 saw state=null with live_nodes=[]
   [junit4]   2>        at 
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:802)
   [junit4]   2>        at 
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:786)
   [junit4]   2>        at 
org.apache.solr.handler.IndexFetcher.getLeaderReplica(IndexFetcher.java:688)
   [junit4]   2>        at 
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:382)
   [junit4]   2>        at 
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:347)
   [junit4]   2>        at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
   [junit4]   2>        at 
org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$12(ReplicationHandler.java:1159)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 1934819 INFO  (Thread-4701) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:45143 45143
   [junit4]   2> 1934820 WARN  (Thread-4701) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        16      
/solr/collections/pull_replica_test_remove_all_writer_replicas/terms/shard1
   [junit4]   2>        13      
/solr/collections/pull_replica_test_create_delete/terms/shard2
   [junit4]   2>        12      
/solr/collections/pull_replica_test_create_delete/terms/shard1
   [junit4]   2>        12      
/solr/collections/pull_replica_test_real_time_get/terms/shard1
   [junit4]   2>        11      /solr/configs/conf
   [junit4]   2>        9       
/solr/collections/pull_replica_test_kill_pull_replica/terms/shard1
   [junit4]   2>        9       /solr/aliases.json
   [junit4]   2>        9       
/solr/collections/pull_replica_test_kill_leader/terms/shard1
   [junit4]   2>        5       /solr/clusterprops.json
   [junit4]   2>        4       /solr/security.json
   [junit4]   2>        4       
/solr/collections/pull_replica_test_add_remove_pull_replica/terms/shard2
   [junit4]   2>        3       
/solr/collections/pull_replica_test_add_remove_pull_replica/terms/shard1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        50      
/solr/collections/pull_replica_test_create_delete/state.json
   [junit4]   2>        23      
/solr/collections/pull_replica_test_kill_leader/state.json
   [junit4]   2>        22      
/solr/collections/pull_replica_test_kill_pull_replica/state.json
   [junit4]   2>        22      
/solr/collections/pull_replica_test_add_remove_pull_replica/state.json
   [junit4]   2>        22      
/solr/collections/pull_replica_test_remove_all_writer_replicas/state.json
   [junit4]   2>        18      
/solr/collections/pull_replica_test_real_time_get/state.json
   [junit4]   2>        9       /solr/clusterprops.json
   [junit4]   2>        9       /solr/clusterstate.json
   [junit4]   2>        2       /solr/autoscaling.json
   [junit4]   2>        2       
/solr/overseer_elect/election/73487496792965124-127.0.0.1:34355_solr-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        9       /solr/live_nodes
   [junit4]   2>        9       /solr/collections
   [junit4]   2>        2       /solr/overseer/queue
   [junit4]   2>        2       /solr/autoscaling/events/.scheduled_maintenance
   [junit4]   2>        2       /solr/autoscaling/events/.auto_add_replicas
   [junit4]   2>        2       /solr/overseer/collection-queue-work
   [junit4]   2>        2       /solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestPullReplica_EEB123C12B731A74-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
{foo=Lucene50(blocksize=128), id=PostingsFormat(name=MockRandom), 
foo_s=Lucene50(blocksize=128)}, 
docValues:{_version_=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=717, 
maxMBSortInHeap=5.912197702545397, sim=RandomSimilarity(queryNorm=false): {}, 
locale=ar-SD, timezone=Europe/Amsterdam
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_144 (64-bit)/cpus=4,threads=1,free=121143344,total=524812288
   [junit4]   2> NOTE: All tests run in this JVM: [TestFieldCache, 
TestImpersonationWithHadoopAuth, DeleteShardTest, TestMaxTokenLenTokenizer, 
TestCloudManagedSchema, ClusterStateTest, TestShardHandlerFactory, 
TestComplexPhraseQParserPlugin, UpdateLogTest, TestManagedSynonymFilterFactory, 
TestBulkSchemaAPI, TestTolerantSearch, TestSearcherReuse, CoreAdminHandlerTest, 
DistributedVersionInfoTest, TestMultiValuedNumericRangeQuery, 
MoreLikeThisHandlerTest, TestSubQueryTransformer, TestAtomicUpdateErrorCases, 
TestLazyCores, TestStandardQParsers, BaseCdcrDistributedZkTest, 
TestCoreDiscovery, TestQueryWrapperFilter, TestMiniSolrCloudClusterSSL, 
TestFileDictionaryLookup, TestLMJelinekMercerSimilarityFactory, 
InfoHandlerTest, ClusterStateUpdateTest, ConcurrentCreateRoutedAliasTest, 
TestUnifiedSolrHighlighter, DistributedSuggestComponentTest, BitVectorTest, 
DateMathParserTest, AnalysisAfterCoreReloadTest, AddReplicaTest, 
LeaderFailureAfterFreshStartTest, URLClassifyProcessorTest, TestInitParams, 
HdfsRestartWhileUpdatingTest, TestLegacyFieldCache, DateFieldTest, TestUtils, 
BasicDistributedZk2Test, TestCloudDeleteByQuery, ResponseLogComponentTest, 
SolrSlf4jReporterTest, SolrIndexMetricsTest, HLLSerializationTest, 
TriggerIntegrationTest, TestHdfsBackupRestoreCore, TestNumericTokenStream, 
TestRandomDVFaceting, TestDFRSimilarityFactory, TestUninvertingReader, 
DistributedFacetPivotLongTailTest, MigrateRouteKeyTest, 
AtomicUpdateProcessorFactoryTest, TimeZoneUtilsTest, TestFiltering, 
TestOverriddenPrefixQueryForCustomFieldType, 
TlogReplayBufferedWhileIndexingTest, ComputePlanActionTest, 
TermVectorComponentTest, TestManagedResource, TestSolrDeletionPolicy2, 
TestLRUStatsCache, PreAnalyzedFieldTest, MinimalSchemaTest, TestCoreAdminApis, 
TestRuleBasedAuthorizationPlugin, NodeMutatorTest, TestUseDocValuesAsStored2, 
TestRestoreCore, TestSchemalessBufferedUpdates, RequestLoggingTest, 
ReplaceNodeTest, StressHdfsTest, SolrXmlInZkTest, 
PreAnalyzedUpdateProcessorTest, TestEmbeddedSolrServerAdminHandler, 
HdfsDirectoryTest, RecoveryZkTest, TestSolr4Spatial2, HttpTriggerListenerTest, 
SearchRateTriggerTest, TestGeoJSONResponseWriter, TestAnalyzedSuggestions, 
TestBinaryResponseWriter, TestPolicyCloud, TestSuggestSpellingConverter, 
V2ApiIntegrationTest, TestDistribIDF, TestTrackingShardHandlerFactory, 
MultiTermTest, ManagedSchemaRoundRobinCloudTest, SolrGraphiteReporterTest, 
CdcrReplicationDistributedZkTest, RegexBoostProcessorTest, 
TestClassicSimilarityFactory, UniqFieldsUpdateProcessorFactoryTest, 
RegexBytesRefFilterTest, TestBadConfig, TestQueryTypes, 
TestLeaderElectionZkExpiry, CircularListTest, TestRealTimeGet, 
TestNamedUpdateProcessors, FileBasedSpellCheckerTest, 
TestTolerantUpdateProcessorCloud, SearchHandlerTest, TestSolrXml, 
PingRequestHandlerTest, HdfsNNFailoverTest, CreateCollectionCleanupTest, 
TestLocalFSCloudBackupRestore, TestSizeLimitedDistributedMap, 
QueryResultKeyTest, WordBreakSolrSpellCheckerTest, 
SpellCheckCollatorWithCollapseTest, CopyFieldTest, 
TestDeleteCollectionOnDownNodes, TestIntervalFaceting, TestDynamicLoading, 
DocExpirationUpdateProcessorFactoryTest, HdfsBasicDistributedZkTest, 
TestStressInPlaceUpdates, ZkCLITest, TestDistributedSearch, TestSolr4Spatial, 
SpellCheckComponentTest, SolrCmdDistributorTest, ConvertedLegacyTest, 
SoftAutoCommitTest, TestHighlightDedupGrouping, TokenizerChainTest, 
TestJettySolrRunner, ConnectionReuseTest, AssignBackwardCompatibilityTest, 
DistribCursorPagingTest, DocValuesNotIndexedTest, HttpPartitionOnCommitTest, 
MoveReplicaHDFSTest, MoveReplicaTest, PeerSyncReplicationTest, 
RecoveryAfterSoftCommitTest, TestAuthenticationFramework, TestCloudConsistency, 
TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestDistributedMap, 
TestDownShardTolerantSearch, TestLeaderElectionWithEmptyReplica, 
TestPrepRecovery, TestPullReplica]
   [junit4] Completed [591/784 (1!)] on J2 in 105.63s, 10 tests, 1 failure, 2 
skipped <<< FAILURES!

[...truncated 50329 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to