Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17914/ Java: 32bit/jdk1.8.0_102 -client -XX:+UseG1GC
18 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Suite timeout exceeded (>= 7200000 msec). Stack Trace: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). at __randomizedtesting.SeedInfo.seed([FF31FA7D3FCCEEA6]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Captured an uncaught exception in thread: Thread[id=4824, name=RMI RenewClean-[127.0.0.1:35112], state=RUNNABLE, group=system] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4824, name=RMI RenewClean-[127.0.0.1:35112], state=RUNNABLE, group=system] Caused by: java.lang.OutOfMemoryError: Java heap space at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:593) at java.lang.Thread.run(Thread.java:745) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Captured an uncaught exception in thread: Thread[id=16166, name=qtp18308286-16166, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=16166, name=qtp18308286-16166, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Caused by: java.lang.OutOfMemoryError: Java heap space FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Captured an uncaught exception in thread: Thread[id=16132, name=qtp26303499-16132, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=16132, name=qtp26303499-16132, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Caused by: java.lang.OutOfMemoryError: Java heap space FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Captured an uncaught exception in thread: Thread[id=11503, name=qtp2285062-11503, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11503, name=qtp2285062-11503, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Caused by: java.lang.OutOfMemoryError: Java heap space FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Captured an uncaught exception in thread: Thread[id=11604, name=org.eclipse.jetty.server.session.HashSessionManager@198a318Timer, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11604, name=org.eclipse.jetty.server.session.HashSessionManager@198a318Timer, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Caused by: java.lang.OutOfMemoryError: Java heap space FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Captured an uncaught exception in thread: Thread[id=11595, name=Scheduler-16944924, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11595, name=Scheduler-16944924, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Caused by: java.lang.OutOfMemoryError: Java heap space FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Captured an uncaught exception in thread: Thread[id=11509, name=org.eclipse.jetty.server.session.HashSessionManager@6602e4Timer, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11509, name=org.eclipse.jetty.server.session.HashSessionManager@6602e4Timer, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Caused by: java.lang.OutOfMemoryError: Java heap space FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Captured an uncaught exception in thread: Thread[id=11541, name=org.eclipse.jetty.server.session.HashSessionManager@18bf919Timer, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11541, name=org.eclipse.jetty.server.session.HashSessionManager@18bf919Timer, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Caused by: java.lang.OutOfMemoryError: Java heap space FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Captured an uncaught exception in thread: Thread[id=11502, name=qtp2285062-11502, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11502, name=qtp2285062-11502, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Caused by: java.lang.OutOfMemoryError: Java heap space FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Captured an uncaught exception in thread: Thread[id=11642, name=OverseerThreadFactory-5967-thread-1-processing-n:127.0.0.1:36157__rkf, state=RUNNABLE, group=Overseer collection creation process.] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11642, name=OverseerThreadFactory-5967-thread-1-processing-n:127.0.0.1:36157__rkf, state=RUNNABLE, group=Overseer collection creation process.] Caused by: java.lang.OutOfMemoryError: Java heap space FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Captured an uncaught exception in thread: Thread[id=11597, name=qtp26303499-11597, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11597, name=qtp26303499-11597, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Caused by: java.lang.OutOfMemoryError: Java heap space FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest Error Message: Captured an uncaught exception in thread: Thread[id=16133, name=qtp26303499-16133, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=16133, name=qtp26303499-16133, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] Caused by: java.lang.OutOfMemoryError: Java heap space FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest Error Message: Suite timeout exceeded (>= 7200000 msec). Stack Trace: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). at __randomizedtesting.SeedInfo.seed([FF31FA7D3FCCEEA6]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest Error Message: Captured an uncaught exception in thread: Thread[id=6652, name=Connection evictor, state=RUNNABLE, group=TGRP-RecoveryZkTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6652, name=Connection evictor, state=RUNNABLE, group=TGRP-RecoveryZkTest] Caused by: java.lang.OutOfMemoryError: Java heap space FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest Error Message: Captured an uncaught exception in thread: Thread[id=6569, name=OverseerCollectionConfigSetProcessor-96664884531036165-127.0.0.1:44296_-n_0000000000, state=RUNNABLE, group=Overseer collection creation process.] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6569, name=OverseerCollectionConfigSetProcessor-96664884531036165-127.0.0.1:44296_-n_0000000000, state=RUNNABLE, group=Overseer collection creation process.] Caused by: java.lang.OutOfMemoryError: Java heap space at __randomizedtesting.SeedInfo.seed([FF31FA7D3FCCEEA6]:0) at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:278) at org.apache.solr.cloud.DistributedQueue.peekElements(DistributedQueue.java:328) at org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:235) at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:227) at java.lang.Thread.run(Thread.java:745) FAILED: org.apache.solr.cloud.CdcrVersionReplicationTest.testCdcrDocVersions Error Message: Test abandoned because suite timeout was reached. Stack Trace: java.lang.Exception: Test abandoned because suite timeout was reached. at __randomizedtesting.SeedInfo.seed([FF31FA7D3FCCEEA6]:0) FAILED: org.apache.solr.cloud.RecoveryZkTest.test Error Message: Test abandoned because suite timeout was reached. Stack Trace: java.lang.Exception: Test abandoned because suite timeout was reached. at __randomizedtesting.SeedInfo.seed([FF31FA7D3FCCEEA6]:0) Build Log: [...truncated 12731 lines...] [junit4] Suite: org.apache.solr.cloud.RecoveryZkTest [junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/init-core-data-001 [junit4] 2> 1001518 INFO (SUITE-RecoveryZkTest-seed#[FF31FA7D3FCCEEA6]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN) [junit4] 2> 1001520 INFO (SUITE-RecoveryZkTest-seed#[FF31FA7D3FCCEEA6]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: / [junit4] 2> 1001521 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER [junit4] 2> 1001521 INFO (Thread-1344) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 1001522 INFO (Thread-1344) [ ] o.a.s.c.ZkTestServer Starting server [junit4] 2> 1001622 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.ZkTestServer start zk server on port:42633 [junit4] 2> 1001631 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml [junit4] 2> 1001633 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml [junit4] 2> 1001634 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 1001635 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt [junit4] 2> 1001637 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt [junit4] 2> 1001638 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml [junit4] 2> 1001639 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml [junit4] 2> 1001640 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json [junit4] 2> 1001641 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt [junit4] 2> 1001642 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt [junit4] 2> 1001643 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt [junit4] 2> 1001724 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/control-001/cores/collection1 [junit4] 2> 1001726 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.e.j.s.Server jetty-9.3.8.v20160314 [junit4] 2> 1001727 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@15f31cf{/,null,AVAILABLE} [junit4] 2> 1001734 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.e.j.s.ServerConnector Started ServerConnector@117d441{SSL,[ssl, http/1.1]}{127.0.0.1:44296} [junit4] 2> 1001734 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.e.j.s.Server Started @1003364ms [junit4] 2> 1001734 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/tempDir-001/control/data, hostContext=/, hostPort=44296, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/control-001/cores} [junit4] 2> 1001735 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0 [junit4] 2> 1001735 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 1001735 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 1001735 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-09-27T15:11:23.023Z [junit4] 2> 1001738 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1001738 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/control-001/solr.xml [junit4] 2> 1001746 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42633/solr [junit4] 2> 1001764 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:44296_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44296_ [junit4] 2> 1001766 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:44296_ ] o.a.s.c.Overseer Overseer (id=96664884531036165-127.0.0.1:44296_-n_0000000000) starting [junit4] 2> 1001770 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:44296_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44296_ [junit4] 2> 1001772 INFO (zkCallback-1042-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1001818 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:44296_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/control-001/cores [junit4] 2> 1001818 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:44296_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1] [junit4] 2> 1001821 INFO (OverseerStateUpdate-96664884531036165-127.0.0.1:44296_-n_0000000000) [n:127.0.0.1:44296_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1 [junit4] 2> 1002835 WARN (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead. [junit4] 2> 1002837 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0 [junit4] 2> 1002873 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test [junit4] 2> 1002987 WARN (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead. [junit4] 2> 1002990 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id [junit4] 2> 1003004 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection [junit4] 2> 1003005 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/control-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/control-001/cores/collection1/data/] [junit4] 2> 1003005 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@d9972a [junit4] 2> 1003006 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=18.943359375, floorSegmentMB=1.6494140625, forceMergeDeletesPctAllowed=13.826855952132028, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0 [junit4] 2> 1003011 WARN (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}} [junit4] 2> 1003023 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 1003023 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 1003024 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 1003024 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 1003024 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=31, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6562690374295047] [junit4] 2> 1003025 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1e6e403[collection1] main] [junit4] 2> 1003026 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1003026 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1003026 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 1003028 INFO (searcherExecutor-3079-thread-1-processing-n:127.0.0.1:44296_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1e6e403[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1003028 INFO (coreLoadExecutor-3078-thread-1-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1546638154075734016 [junit4] 2> 1003034 INFO (coreZkRegister-3071-thread-1-processing-n:127.0.0.1:44296_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. [junit4] 2> 1003034 INFO (coreZkRegister-3071-thread-1-processing-n:127.0.0.1:44296_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync [junit4] 2> 1003034 INFO (coreZkRegister-3071-thread-1-processing-n:127.0.0.1:44296_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:44296/collection1/ [junit4] 2> 1003034 INFO (coreZkRegister-3071-thread-1-processing-n:127.0.0.1:44296_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me [junit4] 2> 1003034 INFO (coreZkRegister-3071-thread-1-processing-n:127.0.0.1:44296_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:44296/collection1/ has no replicas [junit4] 2> 1003037 INFO (coreZkRegister-3071-thread-1-processing-n:127.0.0.1:44296_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:44296/collection1/ shard1 [junit4] 2> 1003188 INFO (coreZkRegister-3071-thread-1-processing-n:127.0.0.1:44296_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary [junit4] 2> 1003324 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1003325 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false [junit4] 2> 1003325 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2 [junit4] 2> 1003403 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-1-001/cores/collection1 [junit4] 2> 1003404 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-1-001 [junit4] 2> 1003406 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.e.j.s.Server jetty-9.3.8.v20160314 [junit4] 2> 1003406 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@52c996{/,null,AVAILABLE} [junit4] 2> 1003411 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.e.j.s.ServerConnector Started ServerConnector@1db9de0{SSL,[ssl, http/1.1]}{127.0.0.1:44571} [junit4] 2> 1003411 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.e.j.s.Server Started @1005040ms [junit4] 2> 1003411 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=44571, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-1-001/cores} [junit4] 2> 1003411 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0 [junit4] 2> 1003411 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 1003411 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 1003412 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-09-27T15:11:24.700Z [junit4] 2> 1003415 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1003415 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-1-001/solr.xml [junit4] 2> 1003427 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42633/solr [junit4] 2> 1003437 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:44571_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1003441 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:44571_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44571_ [junit4] 2> 1003442 INFO (zkCallback-1042-thread-2-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1003443 INFO (zkCallback-1052-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1003443 INFO (zkCallback-1046-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1003495 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:44571_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-1-001/cores [junit4] 2> 1003495 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:44571_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1] [junit4] 2> 1003497 INFO (OverseerStateUpdate-96664884531036165-127.0.0.1:44296_-n_0000000000) [n:127.0.0.1:44296_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1 [junit4] 2> 1003599 INFO (zkCallback-1052-thread-2-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2]) [junit4] 2> 1004508 WARN (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead. [junit4] 2> 1004509 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0 [junit4] 2> 1004523 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test [junit4] 2> 1004594 WARN (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead. [junit4] 2> 1004595 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id [junit4] 2> 1004609 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1 [junit4] 2> 1004609 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-1-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-1-001/cores/collection1/data/] [junit4] 2> 1004609 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@d9972a [junit4] 2> 1004611 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=18.943359375, floorSegmentMB=1.6494140625, forceMergeDeletesPctAllowed=13.826855952132028, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0 [junit4] 2> 1004617 WARN (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}} [junit4] 2> 1004632 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 1004632 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 1004633 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 1004633 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 1004634 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=31, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6562690374295047] [junit4] 2> 1004634 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@490575[collection1] main] [junit4] 2> 1004635 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1004636 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1004636 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 1004638 INFO (searcherExecutor-3090-thread-1-processing-n:127.0.0.1:44571_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@490575[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1004639 INFO (coreLoadExecutor-3089-thread-1-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1546638155764989952 [junit4] 2> 1004645 INFO (coreZkRegister-3084-thread-1-processing-n:127.0.0.1:44571_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. [junit4] 2> 1004645 INFO (coreZkRegister-3084-thread-1-processing-n:127.0.0.1:44571_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync [junit4] 2> 1004645 INFO (coreZkRegister-3084-thread-1-processing-n:127.0.0.1:44571_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:44571/collection1/ [junit4] 2> 1004645 INFO (coreZkRegister-3084-thread-1-processing-n:127.0.0.1:44571_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me [junit4] 2> 1004645 INFO (coreZkRegister-3084-thread-1-processing-n:127.0.0.1:44571_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:44571/collection1/ has no replicas [junit4] 2> 1004648 INFO (coreZkRegister-3084-thread-1-processing-n:127.0.0.1:44571_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:44571/collection1/ shard1 [junit4] 2> 1004750 INFO (zkCallback-1052-thread-2-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2]) [junit4] 2> 1004799 INFO (coreZkRegister-3084-thread-1-processing-n:127.0.0.1:44571_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary [junit4] 2> 1004902 INFO (zkCallback-1052-thread-2-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2]) [junit4] 2> 1005076 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-2-001/cores/collection1 [junit4] 2> 1005077 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-2-001 [junit4] 2> 1005078 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.e.j.s.Server jetty-9.3.8.v20160314 [junit4] 2> 1005079 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1b5bec8{/,null,AVAILABLE} [junit4] 2> 1005083 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.e.j.s.ServerConnector Started ServerConnector@c17c37{SSL,[ssl, http/1.1]}{127.0.0.1:32853} [junit4] 2> 1005084 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.e.j.s.Server Started @1006713ms [junit4] 2> 1005084 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=32853, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-2-001/cores} [junit4] 2> 1005084 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0 [junit4] 2> 1005084 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 1005084 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 1005084 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-09-27T15:11:26.372Z [junit4] 2> 1005087 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1005087 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-2-001/solr.xml [junit4] 2> 1005097 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42633/solr [junit4] 2> 1005106 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:32853_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2) [junit4] 2> 1005109 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:32853_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32853_ [junit4] 2> 1005110 INFO (zkCallback-1046-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1005111 INFO (zkCallback-1042-thread-2-processing-n:127.0.0.1:44296_) [n:127.0.0.1:44296_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1005111 INFO (zkCallback-1052-thread-2-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1005113 INFO (zkCallback-1058-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1005141 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:32853_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-2-001/cores [junit4] 2> 1005142 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [n:127.0.0.1:32853_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1] [junit4] 2> 1005144 INFO (OverseerStateUpdate-96664884531036165-127.0.0.1:44296_-n_0000000000) [n:127.0.0.1:44296_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1 [junit4] 2> 1005246 INFO (zkCallback-1052-thread-2-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3]) [junit4] 2> 1005246 INFO (zkCallback-1058-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3]) [junit4] 2> 1006160 WARN (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead. [junit4] 2> 1006161 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0 [junit4] 2> 1006188 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test [junit4] 2> 1006284 WARN (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead. [junit4] 2> 1006286 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id [junit4] 2> 1006301 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1 [junit4] 2> 1006302 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-2-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_FF31FA7D3FCCEEA6-001/shard-2-001/cores/collection1/data/] [junit4] 2> 1006302 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@d9972a [junit4] 2> 1006303 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=18.943359375, floorSegmentMB=1.6494140625, forceMergeDeletesPctAllowed=13.826855952132028, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0 [junit4] 2> 1006309 WARN (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}} [junit4] 2> 1006327 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 1006327 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 1006328 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 1006328 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 1006330 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=31, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6562690374295047] [junit4] 2> 1006330 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@19add8a[collection1] main] [junit4] 2> 1006331 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1006332 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1006332 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 1006334 INFO (searcherExecutor-3101-thread-1-processing-n:127.0.0.1:32853_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@19add8a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1006335 INFO (coreLoadExecutor-3100-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1546638157543374848 [junit4] 2> 1006339 INFO (coreZkRegister-3095-thread-1-processing-n:127.0.0.1:32853_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1 [junit4] 2> 1006339 INFO (updateExecutor-1055-thread-1-processing-n:127.0.0.1:32853_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery [junit4] 2> 1006340 INFO (recoveryExecutor-1056-thread-1-processing-n:127.0.0.1:32853_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true [junit4] 2> 1006340 INFO (recoveryExecutor-1056-thread-1-processing-n:127.0.0.1:32853_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]] [junit4] 2> 1006340 INFO (recoveryExecutor-1056-thread-1-processing-n:127.0.0.1:32853_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1] [junit4] 2> 1006340 INFO (recoveryExecutor-1056-thread-1-processing-n:127.0.0.1:32853_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null} [junit4] 2> 1006340 INFO (recoveryExecutor-1056-thread-1-processing-n:127.0.0.1:32853_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [https://127.0.0.1:44571/collection1/] and I am [https://127.0.0.1:32853/collection1/] [junit4] 2> 1006343 INFO (recoveryExecutor-1056-thread-1-processing-n:127.0.0.1:32853_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:44571]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:32853_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true] [junit4] 2> 1006443 INFO (zkCallback-1058-thread-1-processing-n:127.0.0.1:32853_) [n:127.0.0.1:32853_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3]) [junit4] 2> 1006443 INFO (zkCallback-1052-thread-2-processing-n:127.0.0.1:44571_) [n:127.0.0.1:44571_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3]) [junit4] 2> 1006455 INFO (qtp17642293-6587) [n:127.0.0.1:44571_ ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true [junit4] 2> 1006455 INFO (qtp17642293-6587) [n:127.0.0.1:44571_ ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering [junit4] 2> 1006455 INFO (qtp17642293-6587) [n:127.0.0.1:44571_ ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:32853_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"https://127.0.0.1:32853","node_name":"127.0.0.1:32853_","state":"recovering"} [junit4] 2> 1006455 INFO (qtp17642293-6587) [n:127.0.0.1:44571_ ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 0 seconds. [junit4] 2> 1006455 INFO (qtp17642293-6587) [n:127.0.0.1:44571_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:32853_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=0 [junit4] 2> 1006646 INFO (TEST-RecoveryZkTest.test-seed#[FF31FA7D3FCCEEA6]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test [junit4] 2> 1006755 INFO (qtp13409707-6554) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-0 (1546638157981679616)]} 0 3 [junit4] 2> 1006783 INFO (qtp13409707-6548) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-0 (1546638158012088320)]} 0 1 [junit4] 2> 1006876 INFO (qtp19221839-6624) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-0 (1546638157993213952)]} 0 1 [junit4] 2> 1006877 INFO (qtp17642293-6594) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-0 (1546638157993213952)]} 0 113 [junit4] 2> 1006884 INFO (qtp13409707-6553) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-1 (1546638158117994496)]} 0 1 [junit4] 2> 1006897 INFO (qtp19221839-6624) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-1 (1546638158127431680)]} 0 0 [junit4] 2> 1006898 INFO (qtp17642293-6594) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-1 (1546638158127431680)]} 0 6 [junit4] 2> 1006905 INFO (qtp13409707-6554) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-2 (1546638158140014592)]} 0 1 [junit4] 2> 1006917 INFO (qtp19221839-6624) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-2 (1546638158146306048)]} 0 0 [junit4] 2> 1006917 INFO (qtp19221839-6619) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-0 (1546638158019428352)]} 0 0 [junit4] 2> 1006917 INFO (qtp17642293-6594) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-2 (1546638158146306048)]} 0 7 [junit4] 2> 1006918 INFO (qtp17642293-6592) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-0 (1546638158019428352)]} 0 128 [junit4] 2> 1006924 INFO (qtp13409707-6549) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-1 (1546638158158888960)]} 0 1 [junit4] 2> 1006926 INFO (qtp13409707-6551) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-3 (1546638158159937536)]} 0 3 [junit4] 2> 1006935 INFO (qtp19221839-6619) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-1 (1546638158167277568)]} 0 0 [junit4] 2> 1006937 INFO (qtp17642293-6594) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-1 (1546638158167277568)]} 0 7 [junit4] 2> 1006938 INFO (qtp19221839-6623) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-3 (1546638158168326144)]} 0 0 [junit4] 2> 1006938 INFO (qtp17642293-6588) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-3 (1546638158168326144)]} 0 7 [junit4] 2> 1006943 INFO (qtp13409707-6551) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-2 (1546638158179860480)]} 0 0 [junit4] 2> 1006944 INFO (qtp13409707-6548) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-4 (1546638158180909056)]} 0 0 [junit4] 2> 1006955 INFO (qtp19221839-6620) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-2 (1546638158187200512)]} 0 0 [junit4] 2> 1006956 INFO (qtp17642293-6588) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-2 (1546638158187200512)]} 0 6 [junit4] 2> 1006956 INFO (qtp19221839-6619) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-4 (1546638158188249088)]} 0 0 [junit4] 2> 1006957 INFO (qtp17642293-6590) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-4 (1546638158188249088)]} 0 7 [junit4] 2> 1006962 INFO (qtp13409707-6551) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-3 (1546638158198734848)]} 0 1 [junit4] 2> 1006963 INFO (qtp13409707-6549) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-2 (-1546638158200832000)]} 0 0 [junit4] 2> 1006972 INFO (qtp19221839-6620) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{delete=[1-2 (-1546638158206074880)]} 0 0 [junit4] 2> 1006972 INFO (qtp17642293-6590) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[1-2 (-1546638158206074880)]} 0 5 [junit4] 2> 1006973 INFO (qtp19221839-6624) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-3 (1546638158207123456)]} 0 0 [junit4] 2> 1006974 INFO (qtp17642293-6593) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-3 (1546638158207123456)]} 0 6 [junit4] 2> 1006978 INFO (qtp13409707-6551) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-5 (1546638158217609216)]} 0 0 [junit4] 2> 1006979 INFO (qtp13409707-6548) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-4 (1546638158218657792)]} 0 0 [junit4] 2> 1006990 INFO (qtp19221839-6624) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-4 (1546638158224949248)]} 0 0 [junit4] 2> 1006990 INFO (qtp19221839-6625) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-5 (1546638158224949249)]} 0 0 [junit4] 2> 1006990 INFO (qtp17642293-6594) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-4 (1546638158224949248)]} 0 6 [junit4] 2> 1006990 INFO (qtp17642293-6590) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-5 (1546638158224949249)]} 0 5 [junit4] 2> 1006994 INFO (qtp13409707-6548) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-2 (-1546638158234386432)]} 0 0 [junit4] 2> 1006996 INFO (qtp13409707-6555) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-6 (1546638158235435008)]} 0 1 [junit4] 2> 1007005 INFO (qtp19221839-6625) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{delete=[2-2 (-1546638158241726465)]} 0 0 [junit4] 2> 1007006 INFO (qtp17642293-6594) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[2-2 (-1546638158241726465)]} 0 5 [junit4] 2> 1007007 INFO (qtp19221839-6624) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-6 (1546638158241726464)]} 0 0 [junit4] 2> 1007007 INFO (qtp17642293-6588) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-6 (1546638158241726464)]} 0 6 [junit4] 2> 1007012 INFO (qtp13409707-6548) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-5 (1546638158252212224)]} 0 0 [junit4] 2> 1007013 INFO (qtp13409707-6550) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-4 (-1546638158254309376)]} 0 0 [junit4] 2> 1007021 INFO (qtp19221839-6621) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-5 (1546638158257455104)]} 0 0 [junit4] 2> 1007022 INFO (qtp17642293-6594) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-5 (1546638158257455104)]} 0 6 [junit4] 2> 1007023 INFO (qtp19221839-6618) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{delete=[1-4 (-1546638158259552256)]} 0 0 [junit4] 2> 1007023 INFO (qtp17642293-6590) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[1-4 (-1546638158259552256)]} 0 5 [junit4] 2> 1007026 INFO (qtp13409707-6551) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-6 (1546638158267940864)]} 0 0 [junit4] 2> 1007029 INFO (qtp13409707-6550) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-7 (1546638158270038016)]} 0 1 [junit4] 2> 1007051 INFO (qtp19221839-6618) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-6 (1546638158273183744)]} 0 0 [junit4] 2> 1007052 INFO (qtp17642293-6594) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-6 (1546638158273183744)]} 0 21 [junit4] 2> 1007055 INFO (qtp19221839-6621) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-7 (1546638158292058112)]} 0 0 [junit4] 2> 1007055 INFO (qtp17642293-6587) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-7 (1546638158292058112)]} 0 6 [junit4] 2> 1007056 INFO (qtp13409707-6549) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-4 (-1546638158299398144)]} 0 0 [junit4] 2> 1007060 INFO (qtp13409707-6548) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-5 (-1546638158303592448)]} 0 0 [junit4] 2> 1007066 INFO (qtp19221839-6623) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{delete=[2-4 (-1546638158305689600)]} 0 0 [junit4] 2> 1007067 INFO (qtp17642293-6594) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[2-4 (-1546638158305689600)]} 0 4 [junit4] 2> 1007069 INFO (qtp19221839-6624) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{delete=[1-5 (-1546638158308835328)]} 0 1 [junit4] 2> 1007070 INFO (qtp17642293-6592) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[1-5 (-1546638158308835328)]} 0 5 [junit4] 2> 1007073 INFO (qtp13409707-6553) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-7 (1546638158316175360)]} 0 0 [junit4] 2> 1007074 INFO (qtp13409707-6551) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-8 (1546638158318272512)]} 0 0 [junit4] 2> 1007082 INFO (qtp19221839-6624) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-7 (1546638158321418240)]} 0 0 [junit4] 2> 1007082 INFO (qtp17642293-6590) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-7 (1546638158321418240)]} 0 5 [junit4] 2> 1007084 INFO (qtp19221839-6624) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-8 (1546638158323515392)]} 0 0 [junit4] 2> 1007085 INFO (qtp17642293-6594) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-8 (1546638158323515392)]} 0 5 [junit4] 2> 1007086 INFO (qtp13409707-6553) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-5 (-1546638158330855424)]} 0 0 [junit4] 2> 1007089 INFO (qtp13409707-6548) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-9 (1546638158334001152)]} 0 0 [junit4] 2> 1007095 INFO (qtp19221839-6620) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{delete=[2-5 (-1546638158336098304)]} 0 0 [junit4] 2> 1007096 INFO (qtp17642293-6590) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[2-5 (-1546638158336098304)]} 0 5 [junit4] 2> 1007098 INFO (qtp19221839-6624) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-9 (1546638158339244032)]} 0 0 [junit4] 2> 1007098 INFO (qtp17642293-6593) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-9 (1546638158339244032)]} 0 4 [junit4] 2> 1007100 INFO (qtp13409707-6550) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-8 (1546638158344486912)]} 0 0 [junit4] 2> 1007102 INFO (qtp13409707-6555) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-6 (-1546638158347632640)]} 0 0 [junit4] 2> 1007109 INFO (qtp19221839-6618) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-8 (1546638158349729792)]} 0 0 [junit4] 2> 1007109 INFO (qtp17642293-6590) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-8 (1546638158349729792)]} 0 5 [junit4] 2> 1007110 INFO (qtp19221839-6624) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{delete=[1-6 (-1546638158352875520)]} 0 0 [junit4] 2> 1007110 INFO (qtp17642293-6587) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[1-6 (-1546638158352875520)]} 0 4 [junit4] 2> 1007113 INFO (qtp13409707-6549) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-9 (1546638158359166976)]} 0 0 [junit4] 2> 1007115 INFO (qtp13409707-6550) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-10 (1546638158360215552)]} 0 0 [junit4] 2> 1007123 INFO (qtp19221839-6620) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-9 (1546638158365458432)]} 0 0 [junit4] 2> 1007123 INFO (qtp17642293-6594) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-9 (1546638158365458432)]} 0 4 [junit4] 2> 1007124 INFO (qtp19221839-6621) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-10 (1546638158366507008)]} 0 0 [junit4] 2> 1007124 INFO (qtp17642293-6590) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-10 (1546638158366507008)]} 0 4 [junit4] 2> 1007128 INFO (qtp13409707-6550) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-6 (-1546638158373847040)]} 0 0 [junit4] 2> 1007129 INFO (qtp13409707-6554) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-11 (1546638158374895616)]} 0 1 [junit4] 2> 1007137 INFO (qtp19221839-6625) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{delete=[2-6 (-1546638158380138496)]} 0 0 [junit4] 2> 1007137 INFO (qtp19221839-6619) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-11 (1546638158381187072)]} 0 0 [junit4] 2> 1007137 INFO (qtp17642293-6593) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[2-6 (-1546638158380138496)]} 0 4 [junit4] 2> 1007137 INFO (qtp17642293-6588) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-11 (1546638158381187072)]} 0 4 [junit4] 2> 1007142 INFO (qtp13409707-6554) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-10 (1546638158389575680)]} 0 0 [junit4] 2> 1007143 INFO (qtp13409707-6551) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-12 (1546638158389575681)]} 0 0 [junit4] 2> 1007151 INFO (qtp19221839-6624) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-10 (1546638158394818560)]} 0 0 [junit4] 2> 1007151 INFO (qtp19221839-6619) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-12 (1546638158394818561)]} 0 0 [junit4] 2> 1007151 INFO (qtp17642293-6587) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-10 (1546638158394818560)]} 0 4 [junit4] 2> 1007151 INFO (qtp17642293-6593) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-12 (1546638158394818561)]} 0 4 [junit4] 2> 1007157 INFO (qtp13409707-6551) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-11 (1546638158404255744)]} 0 0 [junit4] 2> 1007157 INFO (qtp13409707-6548) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-13 (1546638158404255745)]} 0 0 [junit4] 2> 1007165 INFO (qtp19221839-6619) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-11 (1546638158408450048)]} 0 1 [junit4] 2> 1007166 INFO (qtp17642293-6593) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-11 (1546638158408450048)]} 0 5 [junit4] 2> 1007166 INFO (qtp19221839-6623) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-13 (1546638158409498624)]} 0 0 [junit4] 2> 1007167 INFO (qtp17642293-6588) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-13 (1546638158409498624)]} 0 5 [junit4] 2> 1007170 INFO (qtp13409707-6551) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-12 (1546638158417887232)]} 0 0 [junit4] 2> 1007171 INFO (qtp13409707-6554) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-14 (1546638158419984384)]} 0 0 [junit4] 2> 1007179 INFO (qtp19221839-6625) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[2-12 (1546638158423130112)]} 0 0 [junit4] 2> 1007179 INFO (qtp19221839-6620) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:44571/collection1/&wt=javabin&version=2}{add=[1-14 (1546638158424178688)]} 0 0 [junit4] 2> 1007180 INFO (qtp17642293-6588) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-12 (1546638158423130112)]} 0 5 [junit4] 2> 1007180 INFO (qtp17642293-6592) [n:127.0.0.1:44571_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-14 (1546638158424178688)]} 0 5 [junit4] 2> 1007184 INFO (qtp13409707-6554) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-13 (1546638158433615872)]} 0 0 [junit4] 2> 1007184 INFO (qtp13409707-6548) [n:127.0.0.1:44296_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-8 (-1546638158433615873)]} 0 0 [junit4] 2> 1007193 INFO (qtp19221839-6619) [n:127.0.0.1:32853_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.f [...truncated too long message...] ternalFileFieldSortTest, DirectSolrConnectionTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCodecSupport, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestFastWriter, TimeZoneUtilsTest, TestFastOutputStream, OpenExchangeRatesOrgProviderTest, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, CdcrReplicationHandlerTest, CdcrRequestHandlerTest, CdcrVersionReplicationTest] [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CdcrVersionReplicationTest -Dtests.seed=FF31FA7D3FCCEEA6 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-SG -Dtests.timezone=America/Anguilla -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] ERROR 0.00s J0 | CdcrVersionReplicationTest (suite) <<< [junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). [junit4] > at __randomizedtesting.SeedInfo.seed([FF31FA7D3FCCEEA6]:0)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4824, name=RMI RenewClean-[127.0.0.1:35112], state=RUNNABLE, group=system] [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space [junit4] > at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:593) [junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=16166, name=qtp18308286-16166, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=16132, name=qtp26303499-16132, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11503, name=qtp2285062-11503, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11604, name=org.eclipse.jetty.server.session.HashSessionManager@198a318Timer, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11595, name=Scheduler-16944924, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11509, name=org.eclipse.jetty.server.session.HashSessionManager@6602e4Timer, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11541, name=org.eclipse.jetty.server.session.HashSessionManager@18bf919Timer, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11502, name=qtp2285062-11502, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #11: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11642, name=OverseerThreadFactory-5967-thread-1-processing-n:127.0.0.1:36157__rkf, state=RUNNABLE, group=Overseer collection creation process.] [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #12: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11597, name=qtp26303499-11597, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #13: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=16133, name=qtp26303499-16133, state=RUNNABLE, group=TGRP-CdcrVersionReplicationTest] [junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space [junit4] Completed [633/633 (2!)] on J0 in 7366.26s, 1 test, 14 errors <<< FAILURES! [...truncated 1 lines...] [junit4] JVM J0: stdout was not empty, see: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/temp/junit4-J0-20160927_145439_584.sysout [junit4] >>> JVM J0 emitted unexpected output (verbatim) ---- [junit4] java.lang.OutOfMemoryError: Java heap space [junit4] Dumping heap to /home/jenkins/workspace/Lucene-Solr-master-Linux/heapdumps/java_pid17213.hprof ... [junit4] Heap dump file created [569557220 bytes in 19.914 secs] [junit4] <<< JVM J0: EOF ---- [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/temp/junit4-J0-20160927_145439_584.syserr [junit4] >>> JVM J0 emitted unexpected output (verbatim) ---- [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "TEST-CdcrVersionReplicationTest.testCdcrDocVersions-seed#[FF31FA7D3FCCEEA6]-SendThread(127.0.0.1:37575)" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Connection evictor" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Connection evictor" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "TEST-CdcrVersionReplicationTest.testCdcrDocVersions-seed#[FF31FA7D3FCCEEA6]-SendThread(127.0.0.1:37575)" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "TEST-CdcrVersionReplicationTest.testCdcrDocVersions-seed#[FF31FA7D3FCCEEA6]-SendThread(127.0.0.1:37575)" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "TEST-CdcrVersionReplicationTest.testCdcrDocVersions-seed#[FF31FA7D3FCCEEA6]-SendThread(127.0.0.1:37575)" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Connection evictor" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Connection evictor" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Connection evictor" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "OverseerCollectionConfigSetProcessor-96664910230454277-127.0.0.1:36157__rkf-n_0000000000" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp26303499-16168" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "TEST-CdcrVersionReplicationTest.testCdcrDocVersions-seed#[FF31FA7D3FCCEEA6]-SendThread(127.0.0.1:37575)" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread " [junit4] org.eclipse.jetty.server.session.HashSessionManager@1563022Timer" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Connection evictor" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp26303499-16173" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp16254628-11535" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Connection evictor" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Connection evictor" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Scheduler-8298105" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "OverseerHdfsCoreFailoverThread-96664910230454277-127.0.0.1:36157__rkf-n_0000000000" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp16254628-11534" [junit4] [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Connection evictor" [junit4] <<< JVM J0: EOF ---- [...truncated 11047 lines...] BUILD FAILED /home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:763: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:715: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created: * java_pid17181.hprof * java_pid17213.hprof Total time: 175 minutes 0 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts [WARNINGS] Skipping publisher since build result is FAILURE Recording test results Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org