BTW, following the suggestion
from
https://groups.google.com/forum/#!searchin/elasticsearch/shard$20recovery$20fail/elasticsearch/138uWwW-vhM/WIKOhnI-DDYJ
I was able to work around this issue by setting number_of_replicas to 0,
and than after verifying that all the replicas were gone, increasing it
back to 1. the new replicas came up well without any issues.
On Tuesday, February 17, 2015 at 12:15:12 PM UTC+2, Ziv Segal wrote:
>
> +1
>
> I have the same issue. working with 1.4.1
>
> This is my stack trace:
> [2015-02-17 09:36:08,438][WARN ][indices.recovery ] [data_node]
> [deragzicaqwarkgqdotwqecptkfibn-150216][1] recovery from
> [[data_node2][PC-hkqAnSQuPjzFjykEsSg][ip-X-X-X-X][inet[/x.x.x.x:9300]]{max_local_storage_nodes=1,
>
> zone=us-east-1d, master=false}] failed
> org.elasticsearch.transport.RemoteTransportException:
> [data_node2][inet[/x.x.x.x:9300]][internal:index/shard/recovery/start_recovery]
> Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
> [deragzicaqwarkgqdotwqecptkfibn-150216][1] Phase[2] Execution failed
> at
> org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1136)
> at
> org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:654)
> at
> org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:137)
> at
> org.elasticsearch.indices.recovery.RecoverySource.access$2600(RecoverySource.java:74)
> at
> org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:464)
> at
> org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:450)
> at
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.elasticsearch.transport.RemoteTransportException:
> [data_node][inet[/x.x.x.x:9300]][internal:index/shard/recovery/translog_ops]
> Caused by: org.elasticsearch.index.mapper.MapperParsingException: object
> mapping for [json] tried to parse as object, but got EOF, has a concrete
> value been provided to it?
> at
> org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:498)
> at
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:541)
> at
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:490)
> at
> org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:392)
> at
> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:775)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$TranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:433)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$TranslogOperationsRequestHandler.messageReceived(RecoveryTarget.
>
> On Wednesday, February 11, 2015 at 8:27:47 PM UTC+2, Mario Rodriguez wrote:
>>
>> Bumping up.
>> Does anyone have insight into this??
>>
>> Mario
>>
>> On Tuesday, February 10, 2015 at 3:02:12 PM UTC-6, Mario Rodriguez wrote:
>>>
>>>
>>>
>>> On Tuesday, February 10, 2015 at 2:55:59 PM UTC-6, Mario Rodriguez wrote:
>>>>
>>>> I have 7 indices in our ES cluster, and one of the indices has an issue
>>>> that is preventing recovery.
>>>>
>>>> Here is what I am seeing in the log:
>>>>
>>>> [2015-02-10 00:00:02,483][WARN ][indices.recovery ]
>>>> [ES_Server1] [prodcustomer][1] recovery from
>>>> [[ES_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES_Server2][inet[/5.5.5.5:9300]]]
>>>> failed
>>>> org.elasticsearch.transport.RemoteTransportException:
>>>> [ES_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery]
>>>> Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
>>>> [prodcustomer][1] Phase[1] Execution failed
>>>> at
>>>> org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072)
>>>> at
>>>> org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636)
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135)
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72)
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440)
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426)
>>>> at
>>>> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> Caused by:
>>>> org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:
>>>> [prodcustomer][1] Failed to transfer [0] files with total size of [0b]
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280)
>>>> at
>>>> org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068)
>>>> ... 9 more
>>>> Caused by: java.io.EOFException: read past EOF:
>>>> MMapIndexInput(path="D:\ElasticSearchData\new_es_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019")
>>>> at
>>>> org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
>>>> at org.apache.lucene.store.DataInput.readInt(DataInput.java:96)
>>>> at
>>>> org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132)
>>>> at
>>>> org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523)
>>>> at
>>>> org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438)
>>>> at
>>>> org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:433)
>>>> at org.elasticsearch.index.store.Store.getMetadata(Store.java:144)
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145)
>>>> ... 10 more
>>>> [2015-02-10 00:00:02,483][WARN ][indices.cluster ]
>>>> [ES_Server1] [prodcustomer][1] failed to start shard
>>>> org.elasticsearch.indices.recovery.RecoveryFailedException:
>>>> [prodcustomer][1]: Recovery failed from
>>>> [ES_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES_Server2][inet[/5.5.5.5:9300]]
>>>> into
>>>> [ES_Server1][R3ArnIZsSVSsd-VLEaU_Ug][ES_Server1][inet[ES_Server1.verify.local/
>>>> 4.4.4.4:9300]]
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:306)
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoveryTarget.access$200(RecoveryTarget.java:65)
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoveryTarget$3.run(RecoveryTarget.java:184)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> Caused by: org.elasticsearch.transport.RemoteTransportException:
>>>> [ES_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery]
>>>> Caused by: org.elasticsearch.index.engine.RecoveryEngineException:
>>>> [prodcustomer][1] Phase[1] Execution failed
>>>> at
>>>> org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072)
>>>> at
>>>> org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636)
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135)
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72)
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440)
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426)
>>>> at
>>>> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> Caused by:
>>>> org.elasticsearch.indices.recovery.RecoverFilesRecoveryException:
>>>> [prodcustomer][1] Failed to transfer [0] files with total size of [0b]
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280)
>>>> at
>>>> org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068)
>>>> ... 9 more
>>>> Caused by: java.io.EOFException: read past EOF:
>>>> MMapIndexInput(path="D:\ElasticSearchData\new_es_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019")
>>>> at
>>>> org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
>>>> at org.apache.lucene.store.DataInput.readInt(DataInput.java:96)
>>>> at
>>>> org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132)
>>>> at
>>>> org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523)
>>>> at
>>>> org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438)
>>>> at
>>>> org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:433)
>>>> at org.elasticsearch.index.store.Store.getMetadata(Store.java:144)
>>>> at
>>>> org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145)
>>>> ... 10 more
>>>>
>>>>
>>>> Is it as simple as deleting the checksum files?
>>>> Thank you for any insight anyone can provide.
>>>>
>>>> *EDIT We are on version 1.3.1
>>>
>>>
>>>> Mario
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bd771361-be04-4c98-b848-a4322e519f76%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
