Hello, In the past couple of days I've been getting a lot of error messages about corrupted replica shards. The primary shards come up fast after ES process restart but replicas take a long time to come back. Sometimes it takes a few node restarts to 'kick' the nodes to start replica shards.
ES version is 1.3.1 running on CentOS 6.5 hosted at Softlayer. It's a 3-way cluster with 4 logstash feeders hanging off it. Here are the errors; [2014-08-26 15:01:18,682][WARN ][cluster.action.shard ] [log03 / Salvador Dali] [downloader-2014.08][4] received shard failed for [downloader-2014.08][4], node[l9-BQTHSSF-ElhgpPBZ24w], [R], s[INITIALIZING], indexUUID [2vRrb5YlQP6MTVr1chOezg], reason [engine failure, message [corrupted preexisting index][CorruptIndexException[[downloader-2014.08][4] Corrupted index [corrupted_SkU0-ZHZRxivSnGczABb_g] caused by: CorruptIndexException[codec footer mismatch: actual footer=-1676705023 vs expected footer=-1071082520 (resource: NIOFSIndexInput(path="/acc/ES/NBS/nodes/0/indices/downloader-2014.08/4/index/_k9a_es090_0.doc"))]]]] [2014-08-26 15:01:18,682][WARN ][cluster.action.shard ] [log03 / Salvador Dali] [eventlog-2014.06][0] received shard failed for [eventlog-2014.06][0], node[l9-BQTHSSF-ElhgpPBZ24w], [R], s[INITIALIZING], indexUUID [jbvChdRrRB6HTutxPvxMmQ], reason [engine failure, message [corrupted preexisting index][CorruptIndexException[[eventlog-2014.06][0] Corrupted index [corrupted__712QIBQQqafzpBoQwZtcg] caused by: CorruptIndexException[codec footer mismatch: actual footer=0 vs expected footer=-1071082520 (resource: NIOFSIndexInput(path="/acc/ES/NBS/nodes/0/indices/eventlog-2014.06/0/index/_1k4x.nvd"))]]]] [2014-08-26 15:01:18,684][WARN ][cluster.action.shard ] [log03 / Salvador Dali] [eventlog-2014.07][0] received shard failed for [eventlog-2014.07][0], node[l9-BQTHSSF-ElhgpPBZ24w], [R], s[INITIALIZING], indexUUID [T4tTXkPjTaCdSVNTjHfOcg], reason [engine failure, message [corrupted preexisting index][CorruptIndexException[[eventlog-2014.07][0] Corrupted index [corrupted_OzfNRRGyTIq8a1PRhLYG2w] caused by: CorruptIndexException[codec footer mismatch: actual footer=0 vs expected footer=-1071082520 (resource: NIOFSIndexInput(path="/acc/ES/NBS/nodes/0/indices/eventlog-2014.07/0/index/_rqf.nvd"))]]]] ---- Thanks, David -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c0af53fb-6fdd-4624-bf6c-9b9d50081689%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
