Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4402/
Java: 64bit/jdk-9 -XX:-UseCompressedOops -XX:+UseParallelGC
1 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test
Error Message:
Captured an uncaught exception in thread: Thread[id=10465,
name=updateExecutor-2540-thread-3, state=RUNNABLE,
group=TGRP-ChaosMonkeySafeLeaderTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught
exception in thread: Thread[id=10465, name=updateExecutor-2540-thread-3,
state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
at
__randomizedtesting.SeedInfo.seed([1EAD81425A0923DE:96F9BE98F4F54E26]:0)
Caused by: org.apache.solr.common.SolrException: Replica:
http://127.0.0.1:64547/collection1_shard1_replica_n25/ should have been marked
under leader initiated recovery in ZkController but wasn't.
at __randomizedtesting.SeedInfo.seed([1EAD81425A0923DE]:0)
at
org.apache.solr.cloud.LeaderInitiatedRecoveryThread.run(LeaderInitiatedRecoveryThread.java:88)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.base/java.lang.Thread.run(Thread.java:844)
Build Log:
[...truncated 12394 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/init-core-data-001
[junit4] 2> 938941 WARN
(SUITE-ChaosMonkeySafeLeaderTest-seed#[1EAD81425A0923DE]-worker) [ ]
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
[junit4] 2> 938941 INFO
(SUITE-ChaosMonkeySafeLeaderTest-seed#[1EAD81425A0923DE]-worker) [ ]
o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false)
w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 938942 INFO
(SUITE-ChaosMonkeySafeLeaderTest-seed#[1EAD81425A0923DE]-worker) [ ]
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via:
@org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0,
clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth
[junit4] 2> 938942 INFO
(SUITE-ChaosMonkeySafeLeaderTest-seed#[1EAD81425A0923DE]-worker) [ ]
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks:
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 938942 INFO
(SUITE-ChaosMonkeySafeLeaderTest-seed#[1EAD81425A0923DE]-worker) [ ]
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 2> 938943 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 938943 INFO (Thread-2075) [ ] o.a.s.c.ZkTestServer client
port:0.0.0.0/0.0.0.0:0
[junit4] 2> 938943 INFO (Thread-2075) [ ] o.a.s.c.ZkTestServer
Starting server
[junit4] 2> 938946 ERROR (Thread-2075) [ ] o.a.z.s.ZooKeeperServer
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action
on ERROR or SHUTDOWN server state changes
[junit4] 2> 939046 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.ZkTestServer start zk server on port:64523
[junit4] 2> 939056 INFO (zkConnectionManagerCallback-2517-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 939062 INFO (zkConnectionManagerCallback-2519-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 939071 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractZkTestCase put
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
to /configs/conf1/solrconfig.xml
[junit4] 2> 939074 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractZkTestCase put
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml
to /configs/conf1/schema.xml
[junit4] 2> 939077 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractZkTestCase put
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 939079 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractZkTestCase put
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
to /configs/conf1/stopwords.txt
[junit4] 2> 939082 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractZkTestCase put
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt
to /configs/conf1/protwords.txt
[junit4] 2> 939085 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractZkTestCase put
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml
to /configs/conf1/currency.xml
[junit4] 2> 939087 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractZkTestCase put
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
to /configs/conf1/enumsConfig.xml
[junit4] 2> 939090 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractZkTestCase put
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
to /configs/conf1/open-exchange-rates.json
[junit4] 2> 939093 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractZkTestCase put
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 939095 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractZkTestCase put
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
to /configs/conf1/old_synonyms.txt
[junit4] 2> 939097 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractZkTestCase put
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
to /configs/conf1/synonyms.txt
[junit4] 2> 939100 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly
asked otherwise
[junit4] 2> 939529 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp:
٢٠١٧-١١-٢١T١٧:٢٧:٣٧-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 939534 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 939534 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 939534 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 939535 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@76a6ca38{/,null,AVAILABLE}
[junit4] 2> 939538 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.AbstractConnector Started
ServerConnector@7b10dbd2{HTTP/1.1,[http/1.1]}{127.0.0.1:64526}
[junit4] 2> 939538 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.Server Started @941320ms
[junit4] 2> 939538 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/,
solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/tempDir-001/control/data,
hostPort=64526,
coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/control-001/cores,
replicaType=NRT}
[junit4] 2> 939540 ERROR
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 939540 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version
8.0.0
[junit4] 2> 939540 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 939540 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 939540 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2018-01-22T22:34:25.136845Z
[junit4] 2> 939543 INFO (zkConnectionManagerCallback-2521-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 939544 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 939545 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig Loading container configuration from
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/control-001/solr.xml
[junit4] 2> 939551 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay
is ignored
[junit4] 2> 939551 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 939552 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 939556 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64523/solr
[junit4] 2> 939557 INFO (zkConnectionManagerCallback-2525-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 939560 INFO
(zkConnectionManagerCallback-2527-thread-1-processing-n:127.0.0.1:64526_)
[n:127.0.0.1:64526_ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 939722 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64526_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 939723 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64526_ ] o.a.s.c.OverseerElectionContext I am going to be the
leader 127.0.0.1:64526_
[junit4] 2> 939724 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64526_ ] o.a.s.c.Overseer Overseer
(id=99395860204158980-127.0.0.1:64526_-n_0000000000) starting
[junit4] 2> 939743 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64526_ ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:64526_
[junit4] 2> 939745 INFO
(zkCallback-2526-thread-1-processing-n:127.0.0.1:64526_) [n:127.0.0.1:64526_
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 939783 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64526_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.node' (registry 'solr.node') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 939791 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64526_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jvm' (registry 'solr.jvm') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 939791 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64526_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jetty' (registry 'solr.jetty') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 939793 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64526_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
underneath
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/control-001/cores
[junit4] 2> 939806 INFO (zkConnectionManagerCallback-2533-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 939807 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 939808 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64523/solr ready
[junit4] 2> 939812 INFO (qtp967820369-10302) [n:127.0.0.1:64526_ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params
replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:64526_&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 939814 INFO
(OverseerThreadFactory-4369-thread-1-processing-n:127.0.0.1:64526_)
[n:127.0.0.1:64526_ ] o.a.s.c.a.c.CreateCollectionCmd Create collection
control_collection
[junit4] 2> 939922 INFO (qtp967820369-10306) [n:127.0.0.1:64526_ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 939922 INFO (qtp967820369-10306) [n:127.0.0.1:64526_ ]
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4
transient cores
[junit4] 2> 940029 INFO
(zkCallback-2526-thread-1-processing-n:127.0.0.1:64526_) [n:127.0.0.1:64526_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/control_collection/state.json] for collection
[control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 940936 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 940944 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 941021 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 941041 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1'
using configuration from collection control_collection, trusted=true
[junit4] 2> 941041 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.core.control_collection.shard1.replica_n1' (registry
'solr.core.control_collection.shard1.replica_n1') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 941041 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 941041 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore
at
[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/control-001/cores/control_collection_shard1_replica_n1],
dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/control-001/cores/control_collection_shard1_replica_n1/data/]
[junit4] 2> 941045 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy:
minMergeSize=0, mergeFactor=10, maxMergeSize=112599696,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.1]
[junit4] 2> 941079 WARN (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type =
requestHandler,name = /dump,class = DumpRequestHandler,attributes =
{initParams=a, name=/dump, class=DumpRequestHandler},args =
{defaults={a=A,b=B}}}
[junit4] 2> 941128 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.UpdateHandler Using UpdateLog implementation:
org.apache.solr.update.UpdateLog
[junit4] 2> 941128 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 941128 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 941128 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 941130 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy:
minMergeSize=1000, mergeFactor=47, maxMergeSize=9223372036854775807,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.7435954049612848]
[junit4] 2> 941131 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@2004ca78[control_collection_shard1_replica_n1] main]
[junit4] 2> 941136 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf1
[junit4] 2> 941137 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 941137 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 941138 INFO
(searcherExecutor-4372-thread-1-processing-n:127.0.0.1:64526_
x:control_collection_shard1_replica_n1 s:shard1 c:control_collection)
[n:127.0.0.1:64526_ c:control_collection s:shard1
x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore
[control_collection_shard1_replica_n1] Registered new searcher
Searcher@2004ca78[control_collection_shard1_replica_n1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 941138 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using
new clock 1590333765566070784
[junit4] 2> 941144 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 941144 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 941144 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.SyncStrategy Sync replicas to
http://127.0.0.1:64526/control_collection_shard1_replica_n1/
[junit4] 2> 941144 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 941144 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.SyncStrategy
http://127.0.0.1:64526/control_collection_shard1_replica_n1/ has no replicas
[junit4] 2> 941144 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
election, clear LIR
[junit4] 2> 941150 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContext I am the new leader:
http://127.0.0.1:64526/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 941253 INFO
(zkCallback-2526-thread-2-processing-n:127.0.0.1:64526_) [n:127.0.0.1:64526_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/control_collection/state.json] for collection
[control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 941307 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 941309 INFO (qtp967820369-10306) [n:127.0.0.1:64526_
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT}
status=0 QTime=1387
[junit4] 2> 941311 INFO (qtp967820369-10302) [n:127.0.0.1:64526_ ]
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most
30 seconds. Check all shard replicas
[junit4] 2> 941412 INFO
(zkCallback-2526-thread-2-processing-n:127.0.0.1:64526_) [n:127.0.0.1:64526_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/control_collection/state.json] for collection
[control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 941816 INFO
(OverseerCollectionConfigSetProcessor-99395860204158980-127.0.0.1:64526_-n_0000000000)
[n:127.0.0.1:64526_ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may
have disconnected from ZooKeeper
[junit4] 2> 942314 INFO (qtp967820369-10302) [n:127.0.0.1:64526_ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:64526_&wt=javabin&version=2}
status=0 QTime=2502
[junit4] 2> 942317 INFO (zkConnectionManagerCallback-2537-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 942319 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 942320 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64523/solr ready
[junit4] 2> 942320 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection
loss:false
[junit4] 2> 942321 INFO (qtp967820369-10306) [n:127.0.0.1:64526_ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params
replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 942323 INFO
(OverseerThreadFactory-4369-thread-2-processing-n:127.0.0.1:64526_)
[n:127.0.0.1:64526_ ] o.a.s.c.a.c.CreateCollectionCmd Create collection
collection1
[junit4] 2> 942324 WARN
(OverseerThreadFactory-4369-thread-2-processing-n:127.0.0.1:64526_)
[n:127.0.0.1:64526_ ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to
create a collection (collection1) without cores.
[junit4] 2> 942530 INFO (qtp967820369-10306) [n:127.0.0.1:64526_ ]
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most
30 seconds. Check all shard replicas
[junit4] 2> 942530 INFO (qtp967820369-10306) [n:127.0.0.1:64526_ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=2&wt=javabin&version=2}
status=0 QTime=209
[junit4] 2> 942743 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-1-001
of type NRT
[junit4] 2> 942744 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp:
٢٠١٧-١١-٢١T١٧:٢٧:٣٧-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 942744 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 942744 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 942745 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 942745 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@4238b8f0{/,null,AVAILABLE}
[junit4] 2> 942745 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.AbstractConnector Started
ServerConnector@2638e165{HTTP/1.1,[http/1.1]}{127.0.0.1:64535}
[junit4] 2> 942745 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.Server Started @944527ms
[junit4] 2> 942745 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/,
solrconfig=solrconfig.xml,
solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/tempDir-001/jetty1,
hostPort=64535,
coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-1-001/cores,
replicaType=NRT}
[junit4] 2> 942745 ERROR
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 942745 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version
8.0.0
[junit4] 2> 942745 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 942745 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 942746 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2018-01-22T22:34:28.341992Z
[junit4] 2> 942747 INFO (zkConnectionManagerCallback-2539-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 942748 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 942748 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig Loading container configuration from
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-1-001/solr.xml
[junit4] 2> 942751 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay
is ignored
[junit4] 2> 942751 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 942751 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 942753 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64523/solr
[junit4] 2> 942754 INFO (zkConnectionManagerCallback-2543-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 942756 INFO
(zkConnectionManagerCallback-2545-thread-1-processing-n:127.0.0.1:64535_)
[n:127.0.0.1:64535_ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 942764 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64535_ ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (0) -> (1)
[junit4] 2> 942767 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64535_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 942770 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64535_ ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:64535_
[junit4] 2> 942771 INFO
(zkCallback-2526-thread-2-processing-n:127.0.0.1:64526_) [n:127.0.0.1:64526_
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 942772 INFO
(zkCallback-2544-thread-1-processing-n:127.0.0.1:64535_) [n:127.0.0.1:64535_
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 942772 INFO (zkCallback-2536-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 942861 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64535_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.node' (registry 'solr.node') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 942894 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64535_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jvm' (registry 'solr.jvm') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 942894 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64535_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jetty' (registry 'solr.jetty') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 942895 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64535_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
underneath
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-1-001/cores
[junit4] 2> 942921 INFO (qtp967820369-10304) [n:127.0.0.1:64526_ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params
node=127.0.0.1:64535_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 942924 INFO
(OverseerCollectionConfigSetProcessor-99395860204158980-127.0.0.1:64526_-n_0000000000)
[n:127.0.0.1:64526_ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may
have disconnected from ZooKeeper
[junit4] 2> 942925 INFO
(OverseerThreadFactory-4369-thread-3-processing-n:127.0.0.1:64526_)
[n:127.0.0.1:64526_ ] o.a.s.c.a.c.AddReplicaCmd Node Identified
127.0.0.1:64535_ for creating new replica
[junit4] 2> 942931 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 942931 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_ ]
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4
transient cores
[junit4] 2> 943040 INFO
(zkCallback-2544-thread-1-processing-n:127.0.0.1:64535_) [n:127.0.0.1:64535_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 943948 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 8.0.0
[junit4] 2> 943964 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.s.IndexSchema
[collection1_shard1_replica_n21] Schema name=test
[junit4] 2> 944040 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.s.IndexSchema
Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 944062 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.CoreContainer
Creating SolrCore 'collection1_shard1_replica_n21' using configuration from
collection collection1, trusted=true
[junit4] 2> 944062 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.core.collection1.shard1.replica_n21' (registry
'solr.core.collection1.shard1.replica_n21') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 944062 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SolrCore
solr.RecoveryStrategy.Builder
[junit4] 2> 944062 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SolrCore
[[collection1_shard1_replica_n21] ] Opening new SolrCore at
[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-1-001/cores/collection1_shard1_replica_n21],
dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-1-001/cores/collection1_shard1_replica_n21/data/]
[junit4] 2> 944066 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy:
minMergeSize=0, mergeFactor=10, maxMergeSize=112599696,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.1]
[junit4] 2> 944075 WARN (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21]
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type =
requestHandler,name = /dump,class = DumpRequestHandler,attributes =
{initParams=a, name=/dump, class=DumpRequestHandler},args =
{defaults={a=A,b=B}}}
[junit4] 2> 944119 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.UpdateHandler
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 944119 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100
maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 944121 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.CommitTracker
Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 944121 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.CommitTracker
Soft AutoCommit: disabled
[junit4] 2> 944122 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy:
minMergeSize=1000, mergeFactor=47, maxMergeSize=9223372036854775807,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.7435954049612848]
[junit4] 2> 944134 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@29c83a31[collection1_shard1_replica_n21] main]
[junit4] 2> 944136 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf1
[junit4] 2> 944136 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 944137 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 944137 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.UpdateLog
Could not find max version in index or recent updates, using new clock
1590333768710750208
[junit4] 2> 944138 INFO
(searcherExecutor-4383-thread-1-processing-n:127.0.0.1:64535_
x:collection1_shard1_replica_n21 s:shard1 c:collection1) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SolrCore
[collection1_shard1_replica_n21] Registered new searcher
Searcher@29c83a31[collection1_shard1_replica_n21]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 944143 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21]
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 944143 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21]
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 944143 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy
Sync replicas to http://127.0.0.1:64535/collection1_shard1_replica_n21/
[junit4] 2> 944143 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy
Sync Success - now sync replicas to me
[junit4] 2> 944143 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy
http://127.0.0.1:64535/collection1_shard1_replica_n21/ has no replicas
[junit4] 2> 944143 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21]
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
election, clear LIR
[junit4] 2> 944147 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21]
o.a.s.c.ShardLeaderElectionContext I am the new leader:
http://127.0.0.1:64535/collection1_shard1_replica_n21/ shard1
[junit4] 2> 944249 INFO
(zkCallback-2544-thread-1-processing-n:127.0.0.1:64535_) [n:127.0.0.1:64535_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 944306 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.ZkController
I am the leader, no recovery necessary
[junit4] 2> 944308 INFO (qtp1718390160-10350) [n:127.0.0.1:64535_
c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT}
status=0 QTime=1377
[junit4] 2> 944311 INFO (qtp967820369-10304) [n:127.0.0.1:64526_ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={node=127.0.0.1:64535_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2}
status=0 QTime=1390
[junit4] 2> 944411 INFO
(zkCallback-2544-thread-1-processing-n:127.0.0.1:64535_) [n:127.0.0.1:64535_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 944572 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-2-001
of type NRT
[junit4] 2> 944573 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp:
٢٠١٧-١١-٢١T١٧:٢٧:٣٧-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 944574 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 944574 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 944574 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 944575 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@dbc7450{/,null,AVAILABLE}
[junit4] 2> 944575 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.AbstractConnector Started
ServerConnector@5529bc3b{HTTP/1.1,[http/1.1]}{127.0.0.1:64541}
[junit4] 2> 944575 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.Server Started @946357ms
[junit4] 2> 944575 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/,
solrconfig=solrconfig.xml,
solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/tempDir-001/jetty2,
hostPort=64541,
coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-2-001/cores,
replicaType=NRT}
[junit4] 2> 944575 ERROR
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 944576 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version
8.0.0
[junit4] 2> 944576 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 944576 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 944576 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2018-01-22T22:34:30.172300Z
[junit4] 2> 944578 INFO (zkConnectionManagerCallback-2549-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 944579 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 944579 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig Loading container configuration from
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-2-001/solr.xml
[junit4] 2> 944581 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay
is ignored
[junit4] 2> 944581 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 944582 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 944584 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64523/solr
[junit4] 2> 944585 INFO (zkConnectionManagerCallback-2553-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 944588 INFO
(zkConnectionManagerCallback-2555-thread-1-processing-n:127.0.0.1:64541_)
[n:127.0.0.1:64541_ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 944597 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64541_ ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (0) -> (2)
[junit4] 2> 944600 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64541_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 944602 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64541_ ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:64541_
[junit4] 2> 944604 INFO
(zkCallback-2544-thread-1-processing-n:127.0.0.1:64535_) [n:127.0.0.1:64535_
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 944604 INFO
(zkCallback-2526-thread-2-processing-n:127.0.0.1:64526_) [n:127.0.0.1:64526_
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 944605 INFO
(zkCallback-2554-thread-1-processing-n:127.0.0.1:64541_) [n:127.0.0.1:64541_
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 944605 INFO (zkCallback-2536-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 944668 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64541_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.node' (registry 'solr.node') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 944682 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64541_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jvm' (registry 'solr.jvm') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 944682 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64541_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jetty' (registry 'solr.jetty') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 944684 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64541_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
underneath
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-2-001/cores
[junit4] 2> 944721 INFO (qtp967820369-10306) [n:127.0.0.1:64526_ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params
node=127.0.0.1:64541_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 944724 INFO
(OverseerCollectionConfigSetProcessor-99395860204158980-127.0.0.1:64526_-n_0000000000)
[n:127.0.0.1:64526_ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000004 doesn't exist. Requestor may
have disconnected from ZooKeeper
[junit4] 2> 944725 INFO
(OverseerThreadFactory-4369-thread-4-processing-n:127.0.0.1:64526_)
[n:127.0.0.1:64526_ ] o.a.s.c.a.c.AddReplicaCmd Node Identified
127.0.0.1:64541_ for creating new replica
[junit4] 2> 944729 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n23&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 944730 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_ ]
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4
transient cores
[junit4] 2> 944845 INFO
(zkCallback-2554-thread-1-processing-n:127.0.0.1:64541_) [n:127.0.0.1:64541_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [3])
[junit4] 2> 944845 INFO
(zkCallback-2544-thread-1-processing-n:127.0.0.1:64535_) [n:127.0.0.1:64535_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [3])
[junit4] 2> 945751 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 8.0.0
[junit4] 2> 945761 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.s.IndexSchema
[collection1_shard1_replica_n23] Schema name=test
[junit4] 2> 945829 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.s.IndexSchema
Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 945852 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.c.CoreContainer
Creating SolrCore 'collection1_shard1_replica_n23' using configuration from
collection collection1, trusted=true
[junit4] 2> 945852 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.core.collection1.shard1.replica_n23' (registry
'solr.core.collection1.shard1.replica_n23') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 945852 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.c.SolrCore
solr.RecoveryStrategy.Builder
[junit4] 2> 945852 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.c.SolrCore
[[collection1_shard1_replica_n23] ] Opening new SolrCore at
[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-2-001/cores/collection1_shard1_replica_n23],
dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-2-001/cores/collection1_shard1_replica_n23/data/]
[junit4] 2> 945856 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy:
minMergeSize=0, mergeFactor=10, maxMergeSize=112599696,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.1]
[junit4] 2> 945872 WARN (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23]
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type =
requestHandler,name = /dump,class = DumpRequestHandler,attributes =
{initParams=a, name=/dump, class=DumpRequestHandler},args =
{defaults={a=A,b=B}}}
[junit4] 2> 945917 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.u.UpdateHandler
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 945917 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100
maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 945918 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.u.CommitTracker
Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 945918 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.u.CommitTracker
Soft AutoCommit: disabled
[junit4] 2> 945919 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy:
minMergeSize=1000, mergeFactor=47, maxMergeSize=9223372036854775807,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.7435954049612848]
[junit4] 2> 945921 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@5a4b6067[collection1_shard1_replica_n23] main]
[junit4] 2> 945923 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf1
[junit4] 2> 945923 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 945924 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 945924 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.u.UpdateLog
Could not find max version in index or recent updates, using new clock
1590333770584555520
[junit4] 2> 945924 INFO
(searcherExecutor-4394-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.c.SolrCore
[collection1_shard1_replica_n23] Registered new searcher
Searcher@5a4b6067[collection1_shard1_replica_n23]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 945928 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.c.ZkController
Core needs to recover:collection1_shard1_replica_n23
[junit4] 2> 945928 INFO
(updateExecutor-2550-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1) [n:127.0.0.1:64541_
c:collection1 s:shard1 r:core_node24 x:collection1_shard1_replica_n23]
o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 945929 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Starting recovery
process. recoveringAfterStartup=true
[junit4] 2> 945929 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy ######
startupVersions=[[]]
[junit4] 2> 945929 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Begin buffering
updates. core=[collection1_shard1_replica_n23]
[junit4] 2> 945929 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.u.UpdateLog Starting to buffer updates.
FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 945929 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Publishing state of
core [collection1_shard1_replica_n23] as recovering, leader is
[http://127.0.0.1:64535/collection1_shard1_replica_n21/] and I am
[http://127.0.0.1:64541/collection1_shard1_replica_n23/]
[junit4] 2> 945930 INFO (qtp1644258388-10382) [n:127.0.0.1:64541_
c:collection1 s:shard1 x:collection1_shard1_replica_n23] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n23&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT}
status=0 QTime=1200
[junit4] 2> 945931 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Sending prep
recovery command to [http://127.0.0.1:64535]; [WaitForState:
action=PREPRECOVERY&core=collection1_shard1_replica_n21&nodeName=127.0.0.1:64541_&coreNodeName=core_node24&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 945931 INFO (qtp1718390160-10354) [n:127.0.0.1:64535_ ]
o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node24, state:
recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true,
maxTime: 183 s
[junit4] 2> 945931 INFO (qtp1718390160-10354) [n:127.0.0.1:64535_ ]
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1,
shard=shard1, thisCore=collection1_shard1_replica_n21,
leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true,
currentState=down, localState=active, nodeName=127.0.0.1:64541_,
coreNodeName=core_node24, onlyIfActiveCheckResult=false, nodeProps:
core_node24:{"core":"collection1_shard1_replica_n23","base_url":"http://127.0.0.1:64541","node_name":"127.0.0.1:64541_","state":"down","type":"NRT"}
[junit4] 2> 945931 INFO (qtp967820369-10306) [n:127.0.0.1:64526_ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={node=127.0.0.1:64541_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2}
status=0 QTime=1210
[junit4] 2> 946041 INFO
(zkCallback-2544-thread-1-processing-n:127.0.0.1:64535_) [n:127.0.0.1:64535_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [3])
[junit4] 2> 946041 INFO
(zkCallback-2554-thread-1-processing-n:127.0.0.1:64541_) [n:127.0.0.1:64541_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [3])
[junit4] 2> 946141 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-3-001
of type NRT
[junit4] 2> 946142 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp:
٢٠١٧-١١-٢١T١٧:٢٧:٣٧-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 946143 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 946143 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 946143 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 946143 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@60cd5ae3{/,null,AVAILABLE}
[junit4] 2> 946143 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.AbstractConnector Started
ServerConnector@469e86b9{HTTP/1.1,[http/1.1]}{127.0.0.1:64547}
[junit4] 2> 946143 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.e.j.s.Server Started @947925ms
[junit4] 2> 946143 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/,
solrconfig=solrconfig.xml,
solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/tempDir-001/jetty3,
hostPort=64547,
coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-3-001/cores}
[junit4] 2> 946144 ERROR
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 946144 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version
8.0.0
[junit4] 2> 946144 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 946144 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 946144 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2018-01-22T22:34:31.740245Z
[junit4] 2> 946145 INFO (zkConnectionManagerCallback-2560-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 946147 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 946147 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig Loading container configuration from
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-3-001/solr.xml
[junit4] 2> 946147 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay
is ignored
[junit4] 2> 946147 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 946147 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 946152 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64523/solr
[junit4] 2> 946154 INFO (zkConnectionManagerCallback-2564-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 946157 INFO
(zkConnectionManagerCallback-2566-thread-1-processing-n:127.0.0.1:64547_)
[n:127.0.0.1:64547_ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 946165 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64547_ ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (0) -> (3)
[junit4] 2> 946169 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64547_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 946171 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64547_ ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:64547_
[junit4] 2> 946176 INFO
(zkCallback-2544-thread-1-processing-n:127.0.0.1:64535_) [n:127.0.0.1:64535_
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 946176 INFO
(zkCallback-2526-thread-2-processing-n:127.0.0.1:64526_) [n:127.0.0.1:64526_
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 946176 INFO
(zkCallback-2554-thread-1-processing-n:127.0.0.1:64541_) [n:127.0.0.1:64541_
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 946176 INFO (zkCallback-2536-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 946176 INFO
(zkCallback-2565-thread-1-processing-n:127.0.0.1:64547_) [n:127.0.0.1:64547_
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 946231 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64547_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.node' (registry 'solr.node') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 946242 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64547_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jvm' (registry 'solr.jvm') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 946242 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64547_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jetty' (registry 'solr.jetty') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 946244 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE])
[n:127.0.0.1:64547_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
underneath
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-3-001/cores
[junit4] 2> 946329 INFO (qtp967820369-10304) [n:127.0.0.1:64526_ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params
node=127.0.0.1:64547_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 946332 INFO
(OverseerCollectionConfigSetProcessor-99395860204158980-127.0.0.1:64526_-n_0000000000)
[n:127.0.0.1:64526_ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000006 doesn't exist. Requestor may
have disconnected from ZooKeeper
[junit4] 2> 946333 INFO
(OverseerThreadFactory-4369-thread-5-processing-n:127.0.0.1:64526_)
[n:127.0.0.1:64526_ ] o.a.s.c.a.c.AddReplicaCmd Node Identified
127.0.0.1:64547_ for creating new replica
[junit4] 2> 946337 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n25&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 946338 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_ ]
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4
transient cores
[junit4] 2> 946443 INFO
(zkCallback-2565-thread-1-processing-n:127.0.0.1:64547_) [n:127.0.0.1:64547_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [4])
[junit4] 2> 946443 INFO
(zkCallback-2554-thread-1-processing-n:127.0.0.1:64541_) [n:127.0.0.1:64541_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [4])
[junit4] 2> 946443 INFO
(zkCallback-2544-thread-1-processing-n:127.0.0.1:64535_) [n:127.0.0.1:64535_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [4])
[junit4] 2> 946939 INFO (qtp1718390160-10354) [n:127.0.0.1:64535_ ]
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1,
shard=shard1, thisCore=collection1_shard1_replica_n21,
leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true,
currentState=recovering, localState=active, nodeName=127.0.0.1:64541_,
coreNodeName=core_node24, onlyIfActiveCheckResult=false, nodeProps:
core_node24:{"core":"collection1_shard1_replica_n23","base_url":"http://127.0.0.1:64541","node_name":"127.0.0.1:64541_","state":"recovering","type":"NRT"}
[junit4] 2> 946939 INFO (qtp1718390160-10354) [n:127.0.0.1:64535_ ]
o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node24, state: recovering,
checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 946939 INFO (qtp1718390160-10354) [n:127.0.0.1:64535_ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
params={nodeName=127.0.0.1:64541_&onlyIfLeaderActive=true&core=collection1_shard1_replica_n21&coreNodeName=core_node24&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2}
status=0 QTime=1007
[junit4] 2> 947354 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 8.0.0
[junit4] 2> 947367 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.s.IndexSchema
[collection1_shard1_replica_n25] Schema name=test
[junit4] 2> 947442 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Attempting to
PeerSync from [http://127.0.0.1:64535/collection1_shard1_replica_n21/] -
recoveringAfterStartup=[true]
[junit4] 2> 947443 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.u.PeerSync PeerSync:
core=collection1_shard1_replica_n23 url=http://127.0.0.1:64541 START
replicas=[http://127.0.0.1:64535/collection1_shard1_replica_n21/] nUpdates=100
[junit4] 2> 947443 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.s.IndexSchema
Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 947445 INFO (qtp1718390160-10354) [n:127.0.0.1:64535_
c:collection1 s:shard1 r:core_node22 x:collection1_shard1_replica_n21]
o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0
result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0,
maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 947445 INFO (qtp1718390160-10354) [n:127.0.0.1:64535_
c:collection1 s:shard1 r:core_node22 x:collection1_shard1_replica_n21]
o.a.s.c.S.Request [collection1_shard1_replica_n21] webapp= path=/get
params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2}
status=0 QTime=0
[junit4] 2> 947446 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.u.IndexFingerprint IndexFingerprint
millis:0.0 result:{maxVersionSpecified=9223372036854775807,
maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0,
maxDoc=0}
[junit4] 2> 947446 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.u.PeerSync We are already in sync. No
need to do a PeerSync
[junit4] 2> 947446 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.u.DirectUpdateHandler2 start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 947446 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.u.DirectUpdateHandler2 No uncommitted
changes. Skipping IW.commit.
[junit4] 2> 947446 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 947446 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy PeerSync stage of
recovery was successful.
[junit4] 2> 947446 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Replaying updates
buffered during PeerSync.
[junit4] 2> 947446 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy No replay needed.
[junit4] 2> 947447 INFO
(recoveryExecutor-2551-thread-1-processing-n:127.0.0.1:64541_
x:collection1_shard1_replica_n23 s:shard1 c:collection1 r:core_node24)
[n:127.0.0.1:64541_ c:collection1 s:shard1 r:core_node24
x:collection1_shard1_replica_n23] o.a.s.c.RecoveryStrategy Registering as
Active after recovery.
[junit4] 2> 947468 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.c.CoreContainer
Creating SolrCore 'collection1_shard1_replica_n25' using configuration from
collection collection1, trusted=true
[junit4] 2> 947469 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.core.collection1.shard1.replica_n25' (registry
'solr.core.collection1.shard1.replica_n25') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@586a4690
[junit4] 2> 947469 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.c.SolrCore
solr.RecoveryStrategy.Builder
[junit4] 2> 947469 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.c.SolrCore
[[collection1_shard1_replica_n25] ] Opening new SolrCore at
[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-3-001/cores/collection1_shard1_replica_n25],
dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_1EAD81425A0923DE-001/shard-3-001/cores/collection1_shard1_replica_n25/data/]
[junit4] 2> 947474 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy:
minMergeSize=0, mergeFactor=10, maxMergeSize=112599696,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.1]
[junit4] 2> 947486 WARN (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25]
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type =
requestHandler,name = /dump,class = DumpRequestHandler,attributes =
{initParams=a, name=/dump, class=DumpRequestHandler},args =
{defaults={a=A,b=B}}}
[junit4] 2> 947552 INFO
(zkCallback-2544-thread-1-processing-n:127.0.0.1:64535_) [n:127.0.0.1:64535_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [4])
[junit4] 2> 947552 INFO
(zkCallback-2554-thread-1-processing-n:127.0.0.1:64541_) [n:127.0.0.1:64541_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [4])
[junit4] 2> 947552 INFO
(zkCallback-2565-thread-1-processing-n:127.0.0.1:64547_) [n:127.0.0.1:64547_
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [4])
[junit4] 2> 947562 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.u.UpdateHandler
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 947562 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100
maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 947565 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.u.CommitTracker
Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 947565 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.u.CommitTracker
Soft AutoCommit: disabled
[junit4] 2> 947568 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy:
minMergeSize=1000, mergeFactor=47, maxMergeSize=9223372036854775807,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.7435954049612848]
[junit4] 2> 947575 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@54376204[collection1_shard1_replica_n25] main]
[junit4] 2> 947576 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf1
[junit4] 2> 947577 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 947577 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 947577 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.u.UpdateLog
Could not find max version in index or recent updates, using new clock
1590333772317851648
[junit4] 2> 947578 INFO
(searcherExecutor-4405-thread-1-processing-n:127.0.0.1:64547_
x:collection1_shard1_replica_n25 s:shard1 c:collection1) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.c.SolrCore
[collection1_shard1_replica_n25] Registered new searcher
Searcher@54376204[collection1_shard1_replica_n25]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 947582 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.c.ZkController
Core needs to recover:collection1_shard1_replica_n25
[junit4] 2> 947582 INFO
(updateExecutor-2561-thread-1-processing-n:127.0.0.1:64547_
x:collection1_shard1_replica_n25 s:shard1 c:collection1) [n:127.0.0.1:64547_
c:collection1 s:shard1 r:core_node26 x:collection1_shard1_replica_n25]
o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 947583 INFO
(recoveryExecutor-2562-thread-1-processing-n:127.0.0.1:64547_
x:collection1_shard1_replica_n25 s:shard1 c:collection1 r:core_node26)
[n:127.0.0.1:64547_ c:collection1 s:shard1 r:core_node26
x:collection1_shard1_replica_n25] o.a.s.c.RecoveryStrategy Starting recovery
process. recoveringAfterStartup=true
[junit4] 2> 947583 INFO
(recoveryExecutor-2562-thread-1-processing-n:127.0.0.1:64547_
x:collection1_shard1_replica_n25 s:shard1 c:collection1 r:core_node26)
[n:127.0.0.1:64547_ c:collection1 s:shard1 r:core_node26
x:collection1_shard1_replica_n25] o.a.s.c.RecoveryStrategy ######
startupVersions=[[]]
[junit4] 2> 947583 INFO
(recoveryExecutor-2562-thread-1-processing-n:127.0.0.1:64547_
x:collection1_shard1_replica_n25 s:shard1 c:collection1 r:core_node26)
[n:127.0.0.1:64547_ c:collection1 s:shard1 r:core_node26
x:collection1_shard1_replica_n25] o.a.s.c.RecoveryStrategy Begin buffering
updates. core=[collection1_shard1_replica_n25]
[junit4] 2> 947583 INFO
(recoveryExecutor-2562-thread-1-processing-n:127.0.0.1:64547_
x:collection1_shard1_replica_n25 s:shard1 c:collection1 r:core_node26)
[n:127.0.0.1:64547_ c:collection1 s:shard1 r:core_node26
x:collection1_shard1_replica_n25] o.a.s.u.UpdateLog Starting to buffer updates.
FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 947583 INFO
(recoveryExecutor-2562-thread-1-processing-n:127.0.0.1:64547_
x:collection1_shard1_replica_n25 s:shard1 c:collection1 r:core_node26)
[n:127.0.0.1:64547_ c:collection1 s:shard1 r:core_node26
x:collection1_shard1_replica_n25] o.a.s.c.RecoveryStrategy Publishing state of
core [collection1_shard1_replica_n25] as recovering, leader is
[http://127.0.0.1:64535/collection1_shard1_replica_n21/] and I am
[http://127.0.0.1:64547/collection1_shard1_replica_n25/]
[junit4] 2> 947585 INFO (qtp1711467951-10417) [n:127.0.0.1:64547_
c:collection1 s:shard1 x:collection1_shard1_replica_n25] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n25&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT}
status=0 QTime=1248
[junit4] 2> 947586 INFO
(recoveryExecutor-2562-thread-1-processing-n:127.0.0.1:64547_
x:collection1_shard1_replica_n25 s:shard1 c:collection1 r:core_node26)
[n:127.0.0.1:64547_ c:collection1 s:shard1 r:core_node26
x:collection1_shard1_replica_n25] o.a.s.c.RecoveryStrategy Sending prep
recovery command to [http://127.0.0.1:64535]; [WaitForState:
action=PREPRECOVERY&core=collection1_shard1_replica_n21&nodeName=127.0.0.1:64547_&coreNodeName=core_node26&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 947587 INFO (qtp1718390160-10354)
[...truncated too long message...]
y.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527)
[junit4] 2> at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
[junit4] 2> at
org.eclipse.jetty.server.Server.handle(Server.java:530)
[junit4] 2> at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
[junit4] 2> at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
[junit4] 2> at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
[junit4] 2> at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
[junit4] 2> at
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
[junit4] 2> at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
[junit4] 2> at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
[junit4] 2> at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
[junit4] 2> at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2>
[junit4] 2> 969467 INFO (qtp1718390160-10353) [n:127.0.0.1:64535_ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
params={nodeName=127.0.0.1:64547_&onlyIfLeaderActive=true&core=collection1_shard1_replica_n21&coreNodeName=core_node26&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2}
status=400 QTime=17367
[junit4] 2> 969467 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.ChaosMonkey monkey: stop jetty! 64541
[junit4] 2> 969467 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.ChaosMonkey monkey: stop jetty! 64547
[junit4] 2> 969473 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.ChaosMonkey monkey: stop jetty! 64526
[junit4] 2> 969473 ERROR
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper
server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 969473 INFO
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[1EAD81425A0923DE]) [ ]
o.a.s.c.ZkTestServer connecting to 127.0.0.1:64523 64523
[junit4] 2> 969474 INFO (Thread-2125) [ ] o.a.s.c.ZkTestServer
connecting to 127.0.0.1:64523 64523
[junit4] 2> 969474 WARN (Thread-2125) [ ] o.a.s.c.ZkTestServer Watch
limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/clusterstate.json
[junit4] 2> 2 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/live_nodes
[junit4] 2> 2 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=test
-Dtests.seed=1EAD81425A0923DE -Dtests.slow=true -Dtests.locale=ln-CG
-Dtests.timezone=Pacific/Gambier -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] ERROR 30.5s J1 | ChaosMonkeySafeLeaderTest.test <<<
[junit4] > Throwable #1:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught
exception in thread: Thread[id=10465, name=updateExecutor-2540-thread-3,
state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] > at
__randomizedtesting.SeedInfo.seed([1EAD81425A0923DE:96F9BE98F4F54E26]:0)
[junit4] > Caused by: org.apache.solr.common.SolrException: Replica:
http://127.0.0.1:64547/collection1_shard1_replica_n25/ should have been marked
under leader initiated recovery in ZkController but wasn't.
[junit4] > at
__randomizedtesting.SeedInfo.seed([1EAD81425A0923DE]:0)
[junit4] > at
org.apache.solr.cloud.LeaderInitiatedRecoveryThread.run(LeaderInitiatedRecoveryThread.java:88)
[junit4] > at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
[junit4] > at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] > at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
[junit4] > at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> 969476 INFO
(SUITE-ChaosMonkeySafeLeaderTest-seed#[1EAD81425A0923DE]-worker) [ ]
o.a.s.SolrTestCaseJ4 -------------------------------------------------------
Done waiting for tracked resources to be released
[junit4] 2> Jan 22, 2018 10:34:55 PM
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked
thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70),
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@14886bf7),
locale=ln-CG, timezone=Pacific/Gambier
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 9
(64-bit)/cpus=3,threads=1,free=166222032,total=444596224
[junit4] 2> NOTE: All tests run in this JVM: [TestCollationFieldDocValues,
AnalyticsMergeStrategyTest, TestExceedMaxTermLength,
UninvertDocValuesMergePolicyTest, TestComponentsName, TestJsonRequest,
DistributedExpandComponentTest, TestSolrCoreProperties, HealthCheckHandlerTest,
TestExpandComponent, OverriddenZkACLAndCredentialsProvidersTest,
BasicAuthStandaloneTest, TestBinaryResponseWriter, TestBlobHandler,
TestNumericTerms32, HdfsDirectoryTest, TestManagedStopFilterFactory,
TestAuthorizationFramework, TestMiniSolrCloudClusterSSL, ScriptEngineTest,
TestSort, TestNRTOpen, DocValuesTest, BooleanFieldTest, SimpleFacetsTest,
MinimalSchemaTest, TestExactSharedStatsCache, NotRequiredUniqueKeyTest,
XsltUpdateRequestHandlerTest, ClusterStateUpdateTest,
IndexBasedSpellCheckerTest, TestNoOpRegenerator, HLLUtilTest,
FieldAnalysisRequestHandlerTest, TestDelegationWithHadoopAuth,
TestExportWriter, TestSerializedLuceneMatchVersion, TestDistribStateManager,
SoftAutoCommitTest, TlogReplayBufferedWhileIndexingTest, TestCollectionAPI,
UUIDUpdateProcessorFallbackTest, TestAtomicUpdateErrorCases,
TestRuleBasedAuthorizationPlugin, TestLuceneMatchVersion,
DistributedSpellCheckComponentTest, DOMUtilTest,
DeleteLastCustomShardedReplicaTest, SuggestComponentTest, TestLRUStatsCache,
TestCoreDiscovery, ChaosMonkeyNothingIsSafeTest,
TestWordDelimiterFilterFactory, BadComponentTest, TestPolicyCloud,
RequestHandlersTest, NumberUtilsTest, TestInfoStreamLogging, TestFastWriter,
DisMaxRequestHandlerTest, TestLegacyFieldReuse, TestHashQParserPlugin,
TestQueryWrapperFilter, AutoScalingHandlerTest, CollectionStateFormat2Test,
SystemInfoHandlerTest, TestPHPSerializedResponseWriter, TestNodeAddedTrigger,
ConjunctionSolrSpellCheckerTest, CdcrBidirectionalTest, TestRecoveryHdfs,
TestHdfsCloudBackupRestore, TestReversedWildcardFilterFactory,
CustomHighlightComponentTest, TestFieldCacheSortRandom,
HttpSolrCallGetCoreTest, TestSolrConfigHandlerConcurrent, TestRandomDVFaceting,
TestSubQueryTransformerDistrib, TestCloudManagedSchema,
TestSubQueryTransformer, CustomCollectionTest, TestConfigSetsAPI,
MigrateRouteKeyTest, BasicFunctionalityTest, QueryParsingTest, JsonLoaderTest,
TestSolrCloudWithHadoopAuthPlugin, TestSQLHandlerNonCloud, TestCustomSort,
HttpTriggerListenerTest, UpdateParamsTest, InfoHandlerTest,
TestSchemaSimilarityResource, DistribDocExpirationUpdateProcessorTest,
NodeAddedTriggerTest, TimeZoneUtilsTest, ZkSolrClientTest,
CoreAdminRequestStatusTest, TestSmileRequest, DistributedQueueTest,
SpellPossibilityIteratorTest, CollectionTooManyReplicasTest, TestBadConfig,
CdcrRequestHandlerTest, TestCloudDeleteByQuery, TestScoreJoinQPNoScore,
SegmentsInfoRequestHandlerTest, NoCacheHeaderTest, BadIndexSchemaTest,
RankQueryTest, TestXmlQParserPlugin, TestCorePropertiesReload,
TestNumericRangeQuery32, TestFastLRUCache, TestCloudNestedDocsSort,
PreAnalyzedFieldTest, JvmMetricsTest, VMParamsZkACLAndCredentialsProvidersTest,
OverseerStatusTest, AsyncCallRequestStatusResponseTest, TestStressRecovery,
TestSortByMinMaxFunction, ReplicationFactorTest,
DistributedFacetPivotLargeTest, TestDistributedMissingSort, TestCodecSupport,
PeerSyncReplicationTest, ResponseLogComponentTest, SolrCoreMetricManagerTest,
ChaosMonkeySafeLeaderTest]
[junit4] Completed [259/768 (1!)] on J1 in 31.61s, 1 test, 1 error <<<
FAILURES!
[...truncated 42575 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
