Hi We are using SolrCloud 7.2.1 with 3 node Zookeeper ensemble. We have 92 collection each on avg. having 8 shards and 2 replica with 2 EC2 nodes, with JVM size of 18GB (G1 GC). We need your help with the Issue we faced today: The issue is SolrCloud server went into a degraded collections (for few collections) when the Solr backup and the Solr batch CSV update load happened at the same time as backup. The CSV data load was about ~5 GB per shard/replica. We think this happened after zkClient disconnect happened as noted below. We had to restart Solr to bring it back to normal.
1. Is it not suggested to run backup and Solr batch CSV update large load at the same time? 2. In the past we have seen two CSV batch update load in parallel causes issues, is this also not suggested (this issue is not related to that)? 3. Do you think we should increase Zookeeper timeout? 4. How do we know if we need to up the JVM Max memory, and by how much? 5. We also see that once the Solr goes into degraded collection and recovery failed, it NEVER get back to normal, even after when there is no load. Is this a bug? The GC information and Solr Log below https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjAvMDQvMjkvLS0wMl9zb2xyX2djLmxvZy56aXAtLTEtNDAtMzE=&channel=WEB 2020-04-27 07:34:07.322 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10775ms for sessionid 0x171a6fb51310008 .... 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected SOLR Log Below (Curtailed WARN log) -------------------- 2020-04-27 07:26:45.402 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes) java.io.EOFException at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-04-27 07:26:45.405 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes) java.io.EOFException at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-04-27 07:26:45.406 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes) java.io.EOFException at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-04-27 07:26:45.408 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes) java.io.EOFException at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-04-27 07:26:45.410 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes) java.io.EOFException at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-04-27 07:26:45.412 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes) java.io.EOFException at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-04-27 07:26:45.412 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error fetching file, doing one retry...:org.apache.solr.common.SolrException: Unable to download _1kr_r.liv completely. Downloaded 0!=587 at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-04-27 07:26:45.414 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 bytes) java.io.EOFException at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531) at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-04-27 07:26:45.414 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher Error deleting file: _1kr_r.liv java.nio.file.NoSuchFileException: /data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/index.20200427072258517/_1kr_r.liv at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) at java.nio.file.Files.delete(Files.java:1126) at org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373) at org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335) at org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:119) at org.apache.solr.handler.IndexFetcher$DirectoryFile.delete(IndexFetcher.java:1757) at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1667) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531) at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-04-27 07:26:45.449 WARN (recoveryExecutor-4-thread-699-processing-n:mysolrsever.com:6010_solr x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] o.a.s.h.IndexFetcher File _1kq.cfs did not match. expected checksum is 3934327071 and actual is checksum 3095753427. expected length is 33235371 and actual length is 54180642 2020-04-27 07:26:45.449 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fnm did not match. expected checksum is 138861107 and actual is checksum 188997082. expected length is 107460 and actual length is 107460 2020-04-27 07:26:45.457 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mt_Lucene50_0.tim did not match. expected checksum is 2947323803 and actual is checksum 979431341. expected length is 32176 and actual length is 26089 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.doc did not match. expected checksum is 2087731087 and actual is checksum 3540174119. expected length is 110 and actual length is 110 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-657-processing-n:mysolrsever.com:6010_solr x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is 777834759 and actual is checksum 289576304. expected length is 59980991 and actual length is 6236424 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mt.fdx did not match. expected checksum is 1321210176 and actual is checksum 224636564. expected length is 114 and actual length is 106 2020-04-27 07:26:45.521 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fdt did not match. expected checksum is 3790133021 and actual is checksum 2516047051. expected length is 1477755 and actual length is 1477850 2020-04-27 07:26:45.578 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tim did not match. expected checksum is 3690299979 and actual is checksum 411724865. expected length is 4392270 and actual length is 4396498 2020-04-27 07:26:45.588 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.doc did not match. expected checksum is 937224718 and actual is checksum 1176441880. expected length is 110 and actual length is 110 2020-04-27 07:26:45.733 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.tim did not match. expected checksum is 1429458386 and actual is checksum 4155623862. expected length is 2213909 and actual length is 2213939 2020-04-27 07:26:45.754 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.ReplicationHandler Index fetch failed :org.apache.solr.common.SolrException: Unable to download _1kr_r.liv completely. Downloaded 0!=587 at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531) at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-04-27 07:26:45.755 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.c.RecoveryStrategy Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed. at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:228) at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-04-27 07:26:45.755 ERROR (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1) 2020-04-27 07:26:45.780 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene70_0.dvm did not match. expected checksum is 1474431645 and actual is checksum 1271168044. expected length is 275570 and actual length is 274506 2020-04-27 07:26:45.834 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk.fdx did not match. expected checksum is 4129522944 and actual is checksum 4080519415. expected length is 6626 and actual length is 6626 2020-04-27 07:26:45.843 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.si did not match. expected checksum is 2493366746 and actual is checksum 31587394. expected length is 546 and actual length is 546 2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is 1392813257 and actual is checksum 3455988143. expected length is 103 and actual length is 111 2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tip did not match. expected checksum is 2301226889 and actual is checksum 29696594. expected length is 86459 and actual length is 85786 2020-04-27 07:26:45.882 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvm did not match. expected checksum is 112676849 and actual is checksum 354222195. expected length is 286474 and actual length is 286546 2020-04-27 07:26:45.911 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is 2203892485 and actual is checksum 1165086749. expected length is 107460 and actual length is 106580 2020-04-27 07:26:45.911 WARN (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] o.a.s.h.IndexFetcher File _1gk_2z.liv did not match. expected checksum is 3175370827 and actual is checksum 520836483. expected length is 51396 and actual length is 51404 2020-04-27 07:26:45.912 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em.fdx did not match. expected checksum is 967260408 and actual is checksum 969726862. expected length is 3180 and actual length is 3180 2020-04-27 07:26:45.914 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is 719562508 and actual is checksum 956062095. expected length is 16717 and actual length is 21454 2020-04-27 07:26:45.915 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene50_0.tip did not match. expected checksum is 3833337361 and actual is checksum 493358852. expected length is 43996 and actual length is 44026 2020-04-27 07:26:45.916 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected checksum is 1533628328 and actual is checksum 3735455395. expected length is 110 and actual length is 110 2020-04-27 07:26:45.917 WARN (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvd did not match. expected checksum is 75152145 and actual is checksum 1665090946. expected length is 497241188 and actual length is 497254814 2020-04-27 07:26:45.918 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected checksum is 1349491281 and actual is checksum 3808036883. expected length is 25153 and actual length is 32322 2020-04-27 07:26:45.920 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected checksum is 3820733560 and actual is checksum 2819887160. expected length is 571 and actual length is 725 2020-04-27 07:26:46.034 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected checksum is 1547030244 and actual is checksum 1122196351. expected length is 324313 and actual length is 323553 2020-04-27 07:26:46.170 WARN (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is 751850967 and actual is checksum 1026527313. expected length is 508 and actual length is 508 and actual length is 319133 2020-04-27 07:27:30.567 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected checksum is 3435029878 and actual is checksum 1202327807. expected length is 3337719 and actual length is 3375106 2020-04-27 07:27:30.578 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is 2264443519 and actual is checksum 2299249501. expected length is 104 and actual length is 102 2020-04-27 07:27:30.580 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is 1184027188 and actual is checksum 2810465617. expected length is 508 and actual length is 508 2020-04-27 07:27:30.622 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is 2246788287 and actual is checksum 2682862227. expected length is 92 and actual length is 96 2020-04-27 07:27:30.625 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is 3227520043 and actual is checksum 2778425164. expected length is 107137 and actual length is 107137 2020-04-27 07:27:30.627 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is 1964200539 and actual is checksum 3997289541. expected length is 7398 and actual length is 9042 2020-04-27 07:27:30.630 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected checksum is 2165407577 and actual is checksum 2563543943. expected length is 110 and actual length is 110 2020-04-27 07:27:30.632 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected checksum is 1427836076 and actual is checksum 225462388. expected length is 11162 and actual length is 13623 2020-04-27 07:27:30.634 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected checksum is 2998276982 and actual is checksum 1966428087. expected length is 296 and actual length is 353 2020-04-27 07:27:30.637 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected checksum is 2717609594 and actual is checksum 673086945. expected length is 315491 and actual length is 312531 2020-04-27 07:27:30.640 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is 2885031416 and actual is checksum 762513018. expected length is 508 and actual length is 508 2020-04-27 07:27:30.643 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvd did not match. expected checksum is 602612626 and actual is checksum 2826680041. expected length is 1292092 and actual length is 1655171 2020-04-27 07:27:30.669 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected checksum is 401385741 and actual is checksum 3373755803. expected length is 621 and actual length is 715 2020-04-27 07:27:30.671 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 3708405765 and actual is checksum 1057460545. expected length is 508 and actual length is 508 2020-04-27 07:27:30.674 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected checksum is 2755907323 and actual is checksum 1931314059. expected length is 327125 and actual length is 318737 2020-04-27 07:27:30.677 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is 77505520 and actual is checksum 2602905677. expected length is 107240 and actual length is 107460 2020-04-27 07:27:30.680 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected checksum is 4204960097 and actual is checksum 2276943505. expected length is 110 and actual length is 110 2020-04-27 07:27:30.682 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is 3414743450 and actual is checksum 3890346633. expected length is 18415 and actual length is 21922 2020-04-27 07:27:30.685 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected checksum is 2879829129 and actual is checksum 790212338. expected length is 27860 and actual length is 33235 2020-04-27 07:27:30.687 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is 3356980932 and actual is checksum 1544608917. expected length is 108 and actual length is 112 2020-04-27 07:27:30.690 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected checksum is 1979425912 and actual is checksum 584894070. expected length is 3726110 and actual length is 4689653 2020-04-27 07:27:30.700 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.si did not match. expected checksum is 3754169108 and actual is checksum 527001228. expected length is 508 and actual length is 508 2020-04-27 07:27:30.702 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvd did not match. expected checksum is 3981710860 and actual is checksum 1749011618. expected length is 1938123 and actual length is 33694 2020-04-27 07:27:30.710 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fdx did not match. expected checksum is 1515379146 and actual is checksum 3212722427. expected length is 98 and actual length is 84 2020-04-27 07:27:30.712 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fnm did not match. expected checksum is 1819400085 and actual is checksum 3372088644. expected length is 107020 and actual length is 103887 2020-04-27 07:27:30.715 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.doc did not match. expected checksum is 1401562358 and actual is checksum 2224139078. expected length is 110 and actual length is 110 2020-04-27 07:27:30.718 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tim did not match. expected checksum is 3386701421 and actual is checksum 1434467863. expected length is 15944 and actual length is 685 2020-04-27 07:27:30.720 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz.fdt did not match. expected checksum is 4259543754 and actual is checksum 3203989793. expected length is 10616 and actual length is 439 2020-04-27 07:27:30.722 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tip did not match. expected checksum is 725882371 and actual is checksum 3650374401. expected length is 383 and actual length is 114 2020-04-27 07:27:30.725 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvm did not match. expected checksum is 3717717087 and actual is checksum 3712473868. expected length is 303729 and actual length is 117582 2020-04-27 07:27:30.728 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is 3506718582 and actual is checksum 342046803. expected length is 93 and actual length is 104 2020-04-27 07:27:30.731 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is 2920971663 and actual is checksum 2142622028. expected length is 508 and actual length is 508 2020-04-27 07:27:30.733 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected checksum is 723455769 and actual is checksum 2731538349. expected length is 110 and actual length is 110 2020-04-27 07:27:30.735 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected checksum is 2624342764 and actual is checksum 1283943229. expected length is 11424 and actual length is 26688 2020-04-27 07:27:30.738 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected checksum is 978416968 and actual is checksum 1807767534. expected length is 330 and actual length is 597 2020-04-27 07:27:30.740 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected checksum is 3599774596 and actual is checksum 2764190632. expected length is 318017 and actual length is 311617 2020-04-27 07:27:30.743 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected checksum is 3141314417 and actual is checksum 3847788889. expected length is 1258075 and actual length is 3539994 2020-04-27 07:27:30.749 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is 1404317797 and actual is checksum 2471843701. expected length is 105295 and actual length is 107020 2020-04-27 07:27:30.752 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is 3412288246 and actual is checksum 922481525. expected length is 7580 and actual length is 17733 2020-04-27 07:27:30.754 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.doc did not match. expected checksum is 2675540020 and actual is checksum 1452209674. expected length is 110 and actual length is 110 2020-04-27 07:27:30.757 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tim did not match. expected checksum is 665797990 and actual is checksum 2134373365. expected length is 10715 and actual length is 3418 2020-04-27 07:27:30.759 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fdt did not match. expected checksum is 2006427884 and actual is checksum 527297847. expected length is 7178 and actual length is 2326 2020-04-27 07:27:30.761 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fnm did not match. expected checksum is 200552893 and actual is checksum 3290522141. expected length is 106580 and actual length is 105206 2020-04-27 07:27:30.764 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tip did not match. expected checksum is 1511033560 and actual is checksum 3689083691. expected length is 278 and actual length is 154 2020-04-27 07:27:30.767 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvm did not match. expected checksum is 1435050119 and actual is checksum 6864243. expected length is 323217 and actual length is 338063 2020-04-27 07:27:30.770 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.si did not match. expected checksum is 1107038327 and actual is checksum 3068870234. expected length is 508 and actual length is 508 2020-04-27 07:27:30.772 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1.fdx did not match. expected checksum is 2853806777 and actual is checksum 459866651. expected length is 93 and actual length is 87 2020-04-27 07:27:30.775 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvd did not match. expected checksum is 1356537478 and actual is checksum 2594306289. expected length is 1179215 and actual length is 249458 2020-04-27 07:27:30.780 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is 1785795479 and actual is checksum 65014613. expected length is 106580 and actual length is 106800 2020-04-27 07:27:30.783 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected checksum is 1824666616 and actual is checksum 2857527783. expected length is 507 and actual length is 637 2020-04-27 07:27:30.785 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is 1384446542 and actual is checksum 3981590407. expected length is 13686 and actual length is 18516 2020-04-27 07:27:30.788 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected checksum is 3380549662 and actual is checksum 2002436369. expected length is 317777 and actual length is 319213 2020-04-27 07:27:30.791 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected checksum is 1594954666 and actual is checksum 1892723425. expected length is 110 and actual length is 110 2020-04-27 07:27:30.794 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected checksum is 305271241 and actual is checksum 854278263. expected length is 20522 and actual length is 27836 2020-04-27 07:27:30.796 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is 2220527769 and actual is checksum 2016379681. expected length is 508 and actual length is 508 2020-04-27 07:27:30.799 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is 1648054590 and actual is checksum 137025071. expected length is 100 and actual length is 105 2020-04-27 07:27:30.801 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected checksum is 262113808 and actual is checksum 327562225. expected length is 2731808 and actual length is 3909012 2020-04-27 07:27:30.809 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvd did not match. expected checksum is 2157667750 and actual is checksum 3043505208. expected length is 36906 and actual length is 163443 2020-04-27 07:27:30.812 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fdt did not match. expected checksum is 3456088243 and actual is checksum 1715362430. expected length is 453 and actual length is 1636 2020-04-27 07:27:30.814 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fnm did not match. expected checksum is 4028738113 and actual is checksum 4245480574. expected length is 104876 and actual length is 105515 2020-04-27 07:27:30.817 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.doc did not match. expected checksum is 3120632072 and actual is checksum 1945528118. expected length is 110 and actual length is 110 2020-04-27 07:27:30.819 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tip did not match. expected checksum is 3303166970 and actual is checksum 1740028553. expected length is 114 and actual length is 137 2020-04-27 07:27:30.822 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tim did not match. expected checksum is 126700369 and actual is checksum 162509958. expected length is 700 and actual length is 2398 2020-04-27 07:27:30.825 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.fdx did not match. expected checksum is 3157051968 and actual is checksum 137679212. expected length is 84 and actual length is 86 2020-04-27 07:27:30.827 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvm did not match. expected checksum is 3969090178 and actual is checksum 2364978843. expected length is 127576 and actual length is 266581 2020-04-27 07:27:30.830 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n3.si did not match. expected checksum is 3730892058 and actual is checksum 2734980133. expected length is 508 and actual length is 508 2020-04-27 07:27:30.832 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fnm did not match. expected checksum is 3259266514 and actual is checksum 2051283039. expected length is 104855 and actual length is 106819 2020-04-27 07:27:30.835 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tim did not match. expected checksum is 380153843 and actual is checksum 580580220. expected length is 1822 and actual length is 1737 2020-04-27 07:27:30.838 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.doc did not match. expected checksum is 4094454161 and actual is checksum 421930789. expected length is 110 and actual length is 110 2020-04-27 07:27:30.840 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fdt did not match. expected checksum is 911575783 and actual is checksum 2090965125. expected length is 1244 and actual length is 1201 2020-04-27 07:27:30.842 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tip did not match. expected checksum is 1279063865 and actual is checksum 765869778. expected length is 128 and actual length is 130 2020-04-27 07:27:30.845 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvm did not match. expected checksum is 1689040450 and actual is checksum 627554920. expected length is 223473 and actual length is 230157 2020-04-27 07:27:30.848 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.fdx did not match. expected checksum is 1278110969 and actual is checksum 1444534251. expected length is 86 and actual length is 86 2020-04-27 07:27:30.850 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvd did not match. expected checksum is 1992192696 and actual is checksum 3455180678. expected length is 119844 and actual length is 117860 2020-04-27 07:27:30.853 WARN (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.h.IndexFetcher File _1n4.si did not match. expected checksum is 2776236859 and actual is checksum 2572132550. expected length is 508 and actual length is 508 2020-04-27 07:27:35.866 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1lu.si did not match. expected checksum is 2573424453 and actual is checksum 3612352961. expected length is 546 and actual length is 437 2020-04-27 07:27:35.868 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.cfe did not match. expected checksum is 2592320603 and actual is checksum 1374033039. expected length is 289 and actual length is 289 2020-04-27 07:27:35.872 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.si did not match. expected checksum is 2911257988 and actual is checksum 2011202433. expected length is 437 and actual length is 437 2020-04-27 07:27:35.874 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1kr.cfs did not match. expected checksum is 1304802713 and actual is checksum 454030092. expected length is 7598425 and actual length is 49265169 2020-04-27 07:27:37.234 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.si did not match. expected checksum is 1422458436 and actual is checksum 703149874. expected length is 437 and actual length is 437 2020-04-27 07:27:38.117 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is 1892273984 and actual is checksum 2087644984. expected length is 50710602 and actual length is 10402264 2020-04-27 07:27:38.532 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mo.cfe did not match. expected checksum is 3864394245 and actual is checksum 1846889307. expected length is 289 and actual length is 289 2020-04-27 07:27:38.575 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fdx did not match. expected checksum is 660375856 and actual is checksum 571052523. expected length is 114 and actual length is 84 2020-04-27 07:27:38.590 WARN (recoveryExecutor-2639-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=31744 deletes=0 deleteByQuery=0 errors=0 positionOfStart=2720536} 2020-04-27 07:27:39.259 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvm did not match. expected checksum is 421440641 and actual is checksum 2075160524. expected length is 337361 and actual length is 141820 2020-04-27 07:27:44.881 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvd did not match. expected checksum is 47189924 and actual is checksum 3640637009. expected length is 5235629 and actual length is 48679 2020-04-27 07:27:45.335 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tip did not match. expected checksum is 2305624674 and actual is checksum 704932512. expected length is 693 and actual length is 115 2020-04-27 07:27:45.939 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fdt did not match. expected checksum is 1157089232 and actual is checksum 2388254324. expected length is 23435 and actual length is 673 2020-04-27 07:27:45.941 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.si did not match. expected checksum is 2811885732 and actual is checksum 2854129172. expected length is 508 and actual length is 508 2020-04-27 07:27:45.943 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.doc did not match. expected checksum is 1925626873 and actual is checksum 1921615605. expected length is 110 and actual length is 110 2020-04-27 07:27:45.945 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tim did not match. expected checksum is 3627283142 and actual is checksum 3796132842. expected length is 34596 and actual length is 958 2020-04-27 07:27:45.947 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1m9.fnm did not match. expected checksum is 1291024098 and actual is checksum 2700484857. expected length is 107020 and actual length is 104209 2020-04-27 07:27:45.949 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tip did not match. expected checksum is 2785033346 and actual is checksum 1911387906. expected length is 816 and actual length is 760 2020-04-27 07:27:45.951 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvd did not match. expected checksum is 942596692 and actual is checksum 3039336477. expected length is 5557044 and actual length is 4847564 2020-04-27 07:27:45.974 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fdx did not match. expected checksum is 143536399 and actual is checksum 2962535892. expected length is 116 and actual length is 111 2020-04-27 07:27:45.976 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvm did not match. expected checksum is 3884436961 and actual is checksum 1345841996. expected length is 339801 and actual length is 336809 2020-04-27 07:27:45.980 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fdt did not match. expected checksum is 3902419774 and actual is checksum 3791519088. expected length is 23306 and actual length is 21047 2020-04-27 07:27:45.981 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.si did not match. expected checksum is 45470103 and actual is checksum 235515609. expected length is 508 and actual length is 508 2020-04-27 07:27:45.983 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.doc did not match. expected checksum is 408594907 and actual is checksum 2650340971. expected length is 110 and actual length is 110 2020-04-27 07:27:45.985 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tim did not match. expected checksum is 761408352 and actual is checksum 685575313. expected length is 35256 and actual length is 31644 2020-04-27 07:27:45.987 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mm.fnm did not match. expected checksum is 811204953 and actual is checksum 1244308773. expected length is 107460 and actual length is 107460 2020-04-27 07:27:45.989 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fnm did not match. expected checksum is 3813441918 and actual is checksum 2133371346. expected length is 107460 and actual length is 106360 2020-04-27 07:27:45.991 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fdt did not match. expected checksum is 4263416408 and actual is checksum 3722461284. expected length is 16698 and actual length is 21392 2020-04-27 07:27:45.993 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.doc did not match. expected checksum is 632422377 and actual is checksum 501799146. expected length is 110 and actual length is 110 2020-04-27 07:27:45.994 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tim did not match. expected checksum is 3170165892 and actual is checksum 4152937488. expected length is 25090 and actual length is 32328 2020-04-27 07:27:46.000 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tip did not match. expected checksum is 2679120655 and actual is checksum 3243853760. expected length is 553 and actual length is 726 2020-04-27 07:27:46.001 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvm did not match. expected checksum is 2991034554 and actual is checksum 1980969905. expected length is 321785 and actual length is 322117 2020-04-27 07:27:46.004 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected checksum is 2117172713 and actual is checksum 4064616498. expected length is 3479018 and actual length is 4488695 2020-04-27 07:27:46.017 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is 1613761705 and actual is checksum 4154864516. expected length is 106 and actual length is 111 2020-04-27 07:27:46.019 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is 3059265880 and actual is checksum 531516038. expected length is 508 and actual length is 508 2020-04-27 07:27:46.021 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected checksum is 1804496408 and actual is checksum 2903373400. expected length is 627 and actual length is 621 2020-04-27 07:27:46.023 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 2241359534 and actual is checksum 4198491027. expected length is 508 and actual length is 508 2020-04-27 07:27:46.024 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected checksum is 3097209578 and actual is checksum 1124925868. expected length is 312617 and actual length is 323465 2020-04-27 07:27:46.027 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is 1909140548 and actual is checksum 2397297432. expected length is 106580 and actual length is 107460 2020-04-27 07:27:46.029 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected checksum is 971818792 and actual is checksum 478139506. expected length is 110 and actual length is 110 2020-04-27 07:27:46.031 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is 1304288778 and actual is checksum 1544273325. expected length is 18170 and actual length is 17756 2020-04-27 07:27:46.033 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected checksum is 4033259458 and actual is checksum 4016593800. expected length is 27532 and actual length is 26636 2020-04-27 07:27:46.034 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is 385809858 and actual is checksum 3497071885. expected length is 107 and actual length is 104 2020-04-27 07:27:46.036 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected checksum is 3567656601 and actual is checksum 2800338830. expected length is 3622054 and actual length is 3792928 2020-04-27 07:27:46.051 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is 2282222236 and actual is checksum 3183044106. expected length is 98 and actual length is 108 2020-04-27 07:27:46.053 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is 150588833 and actual is checksum 3492244992. expected length is 508 and actual length is 508 2020-04-27 07:27:46.055 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected checksum is 1457615503 and actual is checksum 4089018914. expected length is 110 and actual length is 110 2020-04-27 07:27:46.125 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected checksum is 3866656431 and actual is checksum 2014845387. expected length is 16763 and actual length is 28481 2020-04-27 07:27:46.128 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected checksum is 3008320208 and actual is checksum 808227383. expected length is 419 and actual length is 659 2020-04-27 07:27:46.130 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected checksum is 2437844680 and actual is checksum 1756584754. expected length is 310713 and actual length is 315009 2020-04-27 07:27:46.133 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected checksum is 3036548497 and actual is checksum 343807338. expected length is 2245614 and actual length is 4011325 2020-04-27 07:27:46.146 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is 4233343814 and actual is checksum 2984252206. expected length is 106580 and actual length is 107460 2020-04-27 07:27:46.148 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is 2222747552 and actual is checksum 3771791266. expected length is 11198 and actual length is 18927 2020-04-27 07:27:46.150 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is 2062419431 and actual is checksum 885929423. expected length is 107460 and actual length is 105632 2020-04-27 07:27:46.152 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected checksum is 1574973164 and actual is checksum 209201565. expected length is 516 and actual length is 138 2020-04-27 07:27:46.154 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is 3606049617 and actual is checksum 3605002010. expected length is 14213 and actual length is 1590 2020-04-27 07:27:46.156 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected checksum is 842753199 and actual is checksum 3527006266. expected length is 316073 and actual length is 279943 2020-04-27 07:27:46.159 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected checksum is 2130892015 and actual is checksum 2031251178. expected length is 110 and actual length is 110 2020-04-27 07:27:46.161 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected checksum is 1435858498 and actual is checksum 3731425121. expected length is 21284 and actual length is 2331 2020-04-27 07:27:46.163 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is 20126402 and actual is checksum 1544152005. expected length is 508 and actual length is 508 2020-04-27 07:27:46.165 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is 1651289033 and actual is checksum 1035172250. expected length is 100 and actual length is 86 2020-04-27 07:27:46.166 WARN (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected checksum is 3180610843 and actual is checksum 2027612627. expected length is 2818109 and actual length is 159550 2020-04-27 07:27:52.815 WARN (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.u.UpdateLog Starting log replay tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard14_replica_n52/data/tlog/tlog.0000000000000000972 refcount=2} active=true starting pos=1471697 inSortedOrder=false 2020-04-27 07:27:53.592 WARN (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.u.UpdateLog Starting log replay tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/tlog/tlog.0000000000000000972 refcount=2} active=true starting pos=211445345 inSortedOrder=false 2020-04-27 07:27:56.965 WARN (recoveryExecutor-2597-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=34889 deletes=0 deleteByQuery=0 errors=0 positionOfStart=1404348} 2020-04-27 07:27:57.329 WARN (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=5597 deletes=0 deleteByQuery=0 errors=0 positionOfStart=211445345} 2020-04-27 07:28:03.537 WARN (recoveryExecutor-2654-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35707 deletes=0 deleteByQuery=0 errors=0 positionOfStart=2740945} 2020-04-27 07:28:07.353 WARN (recoveryExecutor-2629-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard15_replica_n56 s:shard15 c:mycollection r:core_node59) [c:mycollection s:shard15 r:core_node59 x:mycollection_shard15_replica_n56] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35950 deletes=0 deleteByQuery=0 errors=0 positionOfStart=3081052} 2020-04-27 07:28:19.721 WARN (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=38066 deletes=0 deleteByQuery=0 errors=0 positionOfStart=1471697} 2020-04-27 07:33:26.278 WARN (updateExecutor-3-thread-827-processing-n:mysolrsever.com:6010_solr) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.c.RecoveryStrategy Stopping recovery for core=[mycollection_shard10_replica_n36] coreNodeName=[core_node39] 2020-04-27 07:33:30.370 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 1480542754 and actual is checksum 2732545715. expected length is 437 and actual length is 508 2020-04-27 07:33:37.918 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is 1480542754 and actual is checksum 2732545715. expected length is 437 and actual length is 508 2020-04-27 07:33:37.945 WARN (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] o.a.s.h.IndexFetcher File _1me.si did not match. expected checksum is 204474971 and actual is checksum 3417943085. expected length is 437 and actual length is 546 2020-04-27 07:34:07.322 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10775ms for sessionid 0x171a6fb51310008 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None 2020-04-27 07:34:07.426 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected 2020-04-28 07:53:30.910 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n2:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 10528ms for sessionid 0x171a6fb51310008 2020-04-28 07:53:31.015 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None 2020-04-28 07:53:31.015 WARN (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
