Hello,
I noticed one of the replica's in Recover Failed status and after trying to
recreate the replica/restart the node to recover, I see below error from
solr log. The leader for this shard8 replica2 i.e. replica1@server 61
seems to be fine and serving the queries.
What does this indicate "SolrIndexWriter was not closed prior to
finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!" from below log.
2018-09-22 06:00:13.686 ERROR (Finalizer) [ ]
o.a.s.u.SolrIndexWriter Error closing IndexWriter2018-09-22
05:58:13.409 INFO (qtp834133664-21) [c:COLL s:shard8 r:core_node16
x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=5
2018-09-22 05:58:13.686 INFO (qtp834133664-19) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=39
2018-09-22 05:58:19.338 ERROR
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.ReplicationHandler Index
fetch failed :org.apache.solr.common.SolrException: Index fetch failed
:
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:598)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:301)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:400)
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:219)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:471)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:284)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.index.CorruptIndexException: codec header
mismatch: actual header=1997958933 vs expected header=1071082519
(resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica2/data/index/_8ktm.fnm")))
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:196)
at
org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255)
at
org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:117)
at
org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1063)
at
org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1075)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:960)
at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:118)
at
org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:93)
at
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257)
at
org.apache.solr.update.DefaultSolrCoreState.changeWriter(DefaultSolrCoreState.java:220)
at
org.apache.solr.update.DefaultSolrCoreState.openIndexWriter(DefaultSolrCoreState.java:245)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:551)
... 12 more
Suppressed: org.apache.lucene.index.CorruptIndexException:
checksum failed (hardware problem?) : expected=e42fdf3e
actual=c0432e62
(resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica2/data/index/_8ktm.fnm")))
at
org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
at
org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462)
at
org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:171)
... 21 more
2018-09-22 05:58:19.338 ERROR
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Error
while trying to recover:org.apache.solr.common.SolrException:
Replication for recovery failed.
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:222)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:471)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:284)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-09-22 05:58:19.338 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Replay
not started, or was not successful... still buffering updates.
2018-09-22 05:58:19.338 ERROR
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy
Recovery failed - trying again... (0)
2018-09-22 05:58:19.338 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Wait
[2.0] seconds before trying to recover again (attempt=1)
2018-09-22 05:58:23.352 INFO (qtp834133664-20) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=0
2018-09-22 05:58:23.486 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=18
2018-09-22 05:58:23.763 INFO (qtp834133664-22) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=53
2018-09-22 05:58:29.339 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Begin
buffering updates. core=[COLL_shard8_replica2]
2018-09-22 05:58:29.339 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.u.UpdateLog Restarting
buffering. previous=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0
errors=0 positionOfStart=0}
2018-09-22 05:58:29.339 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.u.UpdateLog Starting to
buffer updates. FSUpdateLog{state=BUFFERING,
tlog=tlog{file=/app/solr/data/COLL_shard8_replica2/data/tlog/tlog.0000000000000000000.1612285920588857344
refcount=2}}
2018-09-22 05:58:29.339 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy
Publishing state of core [COLL_shard8_replica2] as recovering, leader
is [http://server61:8080/solr/COLL_shard8_replica1/] and I am
[http://server62:8080/solr/COLL_shard8_replica2/]
2018-09-22 05:58:29.343 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Sending
prep recovery command to [http://server61:8080/solr]; [WaitForState:
action=PREPRECOVERY&core=COLL_shard8_replica1&nodeName=server62:8080_solr&coreNodeName=core_node16&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
2018-09-22 05:58:29.351 INFO
(zkCallback-5-thread-23-processing-n:server62:8080_solr) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/COLL/state.json] for collection [COLL] has occurred
- updating... (live nodes size: [16])
2018-09-22 05:58:31.846 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy
Starting Replication Recovery.
2018-09-22 05:58:31.846 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy
Attempting to replicate from
[http://server61:8080/solr/COLL_shard8_replica1/].
2018-09-22 05:58:32.234 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Master's
generation: 51970
2018-09-22 05:58:32.234 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Master's
version: 1537595911854
2018-09-22 05:58:32.234 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Slave's
generation: 1
2018-09-22 05:58:32.234 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Slave's
version: 0
2018-09-22 05:58:32.234 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Starting
replication process
2018-09-22 05:58:32.290 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Number of
files in latest index in master: 257
2018-09-22 05:58:32.309 ERROR
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.ReplicationHandler Index
fetch failed :org.apache.solr.common.SolrException: Index fetch failed
:
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:598)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:301)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:400)
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:219)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:471)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:284)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.index.CorruptIndexException: codec header
mismatch: actual header=1997958933 vs expected header=1071082519
(resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica2/data/index/_8ktm.fnm")))
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:196)
at
org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255)
at
org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:117)
at
org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1063)
at
org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1075)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:960)
at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:118)
at
org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:93)
at
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257)
at
org.apache.solr.update.DefaultSolrCoreState.changeWriter(DefaultSolrCoreState.java:220)
at
org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(DefaultSolrCoreState.java:229)
at
org.apache.solr.update.DirectUpdateHandler2.newIndexWriter(DirectUpdateHandler2.java:726)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:457)
... 12 more
Suppressed: org.apache.lucene.index.CorruptIndexException:
checksum failed (hardware problem?) : expected=e42fdf3e
actual=c0432e62
(resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica2/data/index/_8ktm.fnm")))
at
org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
at
org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462)
at
org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:171)
... 22 more
2018-09-22 05:58:32.309 ERROR
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Error
while trying to recover:org.apache.solr.common.SolrException:
Replication for recovery failed.
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:222)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:471)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:284)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-09-22 05:58:32.310 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Replay
not started, or was not successful... still buffering updates.
2018-09-22 05:58:32.310 ERROR
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy
Recovery failed - trying again... (1)
2018-09-22 05:58:32.310 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Wait
[4.0] seconds before trying to recover again (attempt=2)
2018-09-22 05:58:33.423 INFO (qtp834133664-15) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=4
2018-09-22 05:58:33.548 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=3
2018-09-22 05:58:33.834 INFO (qtp834133664-19) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=44
2018-09-22 05:58:43.480 INFO (qtp834133664-13) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=2
2018-09-22 05:58:43.624 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=8
2018-09-22 05:58:43.882 INFO (qtp834133664-22) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=23
2018-09-22 05:58:52.310 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Begin
buffering updates. core=[COLL_shard8_replica2]
2018-09-22 05:58:52.310 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.u.UpdateLog Restarting
buffering. previous=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0
errors=0 positionOfStart=2630296}
2018-09-22 05:58:52.310 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.u.UpdateLog Starting to
buffer updates. FSUpdateLog{state=BUFFERING,
tlog=tlog{file=/app/solr/data/COLL_shard8_replica2/data/tlog/tlog.0000000000000000000.1612285920588857344
refcount=2}}
2018-09-22 05:58:52.310 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy
Publishing state of core [COLL_shard8_replica2] as recovering, leader
is [http://server61:8080/solr/COLL_shard8_replica1/] and I am
[http://server62:8080/solr/COLL_shard8_replica2/]
2018-09-22 05:58:52.317 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Sending
prep recovery command to [http://server61:8080/solr]; [WaitForState:
action=PREPRECOVERY&core=COLL_shard8_replica1&nodeName=server62:8080_solr&coreNodeName=core_node16&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
2018-09-22 05:58:52.325 INFO
(zkCallback-5-thread-23-processing-n:server62:8080_solr) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/COLL/state.json] for collection [COLL] has occurred
- updating... (live nodes size: [16])
2018-09-22 05:58:53.538 INFO (qtp834133664-20) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=8
2018-09-22 05:58:53.679 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=4
2018-09-22 05:58:53.983 INFO (qtp834133664-19) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=63
2018-09-22 05:58:54.820 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy
Starting Replication Recovery.
2018-09-22 05:58:54.820 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy
Attempting to replicate from
[http://server61:8080/solr/COLL_shard8_replica1/].
2018-09-22 05:58:54.967 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Master's
generation: 51971
2018-09-22 05:58:54.967 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Master's
version: 1537595934827
2018-09-22 05:58:54.967 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Slave's
generation: 1
2018-09-22 05:58:54.967 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Slave's
version: 0
2018-09-22 05:58:54.967 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Starting
replication process
2018-09-22 05:58:55.018 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Number of
files in latest index in master: 173
2018-09-22 05:58:55.019 INFO
(recoveryExecutor-3-thread-3-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Starting
download (fullCopy=true) to
NRTCachingDirectory(MMapDirectory@/app/solr/data/COLL_shard8_replica2/data/index.20180922055855018
lockFactory=org.apache.lucene.store.NativeFSLockFactory@74255cc4;
maxCacheMB=48.0 maxMergeSizeMB=4.0)
2018-09-22 05:59:03.590 INFO (qtp834133664-15) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=0
2018-09-22 05:59:03.781 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=25
2018-09-22 05:59:04.061 INFO (qtp834133664-22) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=52
2018-09-22 05:59:13.125 INFO (qtp834133664-13) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=42
2018-09-22 05:59:13.672 INFO (qtp834133664-20) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=2
2018-09-22 05:59:13.812 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=9
2018-09-22 05:59:23.229 INFO (qtp834133664-19) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=43
2018-09-22 05:59:23.275 INFO (qtp834133664-15) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=18
2018-09-22 05:59:23.732 INFO (qtp834133664-20) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=3
2018-09-22 05:59:23.915 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=18
2018-09-22 05:59:33.390 INFO (qtp834133664-22) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=89
2018-09-22 05:59:33.778 INFO (qtp834133664-20) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=0
2018-09-22 05:59:33.958 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=19
2018-09-22 05:59:43.461 INFO (qtp834133664-13) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=46
2018-09-22 05:59:43.862 INFO (qtp834133664-20) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=2
2018-09-22 05:59:44.009 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=15
2018-09-22 05:59:53.515 INFO (qtp834133664-19) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=24
2018-09-22 05:59:53.915 INFO (qtp834133664-20) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=4
2018-09-22 05:59:54.075 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=15
2018-09-22 06:00:03.177 INFO (qtp834133664-15) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=21
2018-09-22 06:00:03.596 INFO (qtp834133664-22) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=51
2018-09-22 06:00:03.977 INFO (qtp834133664-20) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=12
2018-09-22 06:00:13.207 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=4
2018-09-22 06:00:13.685 INFO (qtp834133664-13) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=65
2018-09-22 06:00:13.686 ERROR (Finalizer) [ ]
o.a.s.u.SolrIndexWriter SolrIndexWriter was not closed prior to
finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
2018-09-22 06:00:13.686 ERROR (Finalizer) [ ]
o.a.s.u.SolrIndexWriter Error closing IndexWriter
java.lang.NullPointerException
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3414)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3376)
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1126)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1171)
at
org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:278)
at
org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:334)
at java.lang.System$2.invokeFinalize(System.java:1270)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)
2018-09-22 06:00:13.686 ERROR (Finalizer) [ ]
o.a.s.u.SolrIndexWriter SolrIndexWriter was not closed prior to
finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
2018-09-22 06:00:13.687 ERROR (Finalizer) [ ]
o.a.s.u.SolrIndexWriter Error closing IndexWriter
java.lang.NullPointerException
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3414)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3376)
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1126)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1171)
at
org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:278)
at
org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:334)
at java.lang.System$2.invokeFinalize(System.java:1270)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)
2018-09-22 06:00:14.018 INFO (qtp834133664-20) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=7
2018-09-22 06:00:23.270 INFO (qtp834133664-15) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=15
2018-09-22 06:00:23.775 INFO (qtp834133664-19) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=42
2018-09-22 06:00:24.058 INFO (qtp834133664-20) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=0
2018-09-22 06:00:33.139 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=19
2018-09-22 06:00:33.345 INFO (qtp834133664-15) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=24
2018-09-22 06:00:33.844 INFO (qtp834133664-22) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=43
2018-09-22 06:00:43.165 INFO (qtp834133664-20) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=0
2018-09-22 06:00:43.398 INFO (qtp834133664-15) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=8
2018-09-22 06:00:43.915 INFO (qtp834133664-13) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=46
2018-09-22 06:00:53.211 INFO (qtp834133664-21) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=2
2018-09-22 06:00:53.448 INFO (qtp834133664-15) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=0
2018-09-22 06:00:53.987 INFO (qtp834133664-19) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=49
2018-09-22 06:01:03.252 INFO (qtp834133664-20) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request
[COLL_shard8_replica2] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2}
status=0 QTime=0
