I have 7 indices in our ES cluster, and one of the indices has an issue that is preventing recovery.
Here is what I am seeing in the log: [2015-02-10 00:00:02,483][WARN ][indices.recovery ] [ES_Server1] [prodcustomer][1] recovery from [[ES_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES_Server2][inet[/5.5.5.5:9300]]] failed org.elasticsearch.transport.RemoteTransportException: [ES_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [prodcustomer][1] Phase[1] Execution failed at org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072) at org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636) at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135) at org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: [prodcustomer][1] Failed to transfer [0] files with total size of [0b] at org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280) at org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068) ... 9 more Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="D:\ElasticSearchData\new_es_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81) at org.apache.lucene.store.DataInput.readInt(DataInput.java:96) at org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132) at org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523) at org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438) at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:433) at org.elasticsearch.index.store.Store.getMetadata(Store.java:144) at org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145) ... 10 more [2015-02-10 00:00:02,483][WARN ][indices.cluster ] [ES_Server1] [prodcustomer][1] failed to start shard org.elasticsearch.indices.recovery.RecoveryFailedException: [prodcustomer][1]: Recovery failed from [ES_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES_Server2][inet[/5.5.5.5:9300]] into [ES_Server1][R3ArnIZsSVSsd-VLEaU_Ug][ES_Server1][inet[ES_Server1.verify.local/4.4.4.4:9300]] at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:306) at org.elasticsearch.indices.recovery.RecoveryTarget.access$200(RecoveryTarget.java:65) at org.elasticsearch.indices.recovery.RecoveryTarget$3.run(RecoveryTarget.java:184) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.elasticsearch.transport.RemoteTransportException: [ES_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [prodcustomer][1] Phase[1] Execution failed at org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072) at org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636) at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135) at org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: [prodcustomer][1] Failed to transfer [0] files with total size of [0b] at org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280) at org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068) ... 9 more Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="D:\ElasticSearchData\new_es_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81) at org.apache.lucene.store.DataInput.readInt(DataInput.java:96) at org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132) at org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523) at org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438) at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:433) at org.elasticsearch.index.store.Store.getMetadata(Store.java:144) at org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145) ... 10 more Is it as simple as deleting the checksum files? Thank you for any insight anyone can provide. Mario -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6c11de11-2d3b-4ed3-a173-bae4b50edf8c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
