Corrupted Index
Dear all, We are using ES-1.3.7 for our search Application. Sometime back we upgraded from 0.90.5 to 1.3.7. We have 2 master nodes and 3 data nodes. We are getting CorruptedIndexException when Shard initialization is happening. This is the second time we are facing such issue since last update. Last time, only one shard got corrupted, but now almost 15 to 20 shards got corrupted. Each shard has only ~500MB data. log trace : [2015-04-19 22:49:57,552][WARN ][cluster.action.shard ] [Node1] [138][3] received shard failed for [138][3], node[EkvXNBUOTcuEfWo4SG72bA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[138][3] failed to fetch index version after copying it over]; nested: CorruptIndexException[[138][3] Corrupted index [corrupted_gb9JvBzdRQKqkhEeaXFEIA] caused by: CorruptIndexException[checksum failed (hardware problem?) : expected=637c1x actual=gavi2b resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@54916f2a)]]; ]] [2015-04-19 22:49:57,626][WARN ][cluster.action.shard ] [Node1] [138][3] received shard failed for [138][3], node[Q1eAQgNtSJ2BLlMevzRzcA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[138][3] failed recovery]; nested: EngineCreationFailureException[[138][3] failed to open reader on writer]; nested: FileNotFoundException[No such file [_1gy_x.del]]; ]] Thanks in advance Ranjith Venkatesan -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6f1881d0-9f43-4a01-a9e3-9337a1486fed%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
My first corrupted index
I had a disk problem on my development laptop and my ES index is only
barely working. Before restoring a [old] snapshot and just moving forward,
I'd like to learn if this can be recovered.
My first sign of trouble were query errors like this:
{
error: NoShardAvailableActionException[[twitter_profiles][0] null],
status: 503
}
The disk checks out OK now, but the logs below show what happens when the
node starts up.
What's best way to proceed?
Thanks in advance!
--
The details are here http://pastebin.com/BCXe87Ri, but the high points
are:
[2015-03-26 20:36:55,065][INFO ][node ] [Glob] version[
*1.4.2*], pid[10652], build[927caff/2014-12-16T14:11:12Z]
[2015-03-26 20:37:03,101][WARN ][indices.cluster ] [Glob]
[twitter_profiles][0] *failed to start shard*
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[twitter_profiles][0] failed to fetch index version after copying it over
Caused by: org.apache.lucene.index.CorruptIndexException:
[twitter_profiles][0] *Preexisting corrupted index*
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:03,261][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] *sending failed shard* for [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[twitter_profiles][0] failed to fetch
index version after copying it over]; nested:
CorruptIndexException[[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:03,262][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] *received shard failed* for [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[twitter_profiles][0] failed to fetch
index version after copying it over]; nested:
CorruptIndexException[[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:03,262][INFO ][gateway ] [Glob] *recovered
[2] indices into cluster_state*
[2015-03-26 20:37:17,612][WARN ][indices.cluster ] [Glob]
[twitter_profiles][0] *failed to start shard*
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[twitter_profiles][0] *failed to fetch index version after copying it over*
Caused by: org.apache.lucene.index.CorruptIndexException:
[twitter_profiles][0] *Preexisting corrupted index*
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:17,613][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] *sending failed shard for* [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[twitter_profiles][0] failed to fetch
index version after copying it over]; nested:
CorruptIndexException[[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:17,613][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] *received shard failed* for [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[twitter_profiles][0] failed to fetch
index version after copying it over]; nested:
CorruptIndexException[[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
Badly corrupted index
Our storage server threw it's toys of the pram. We got it back, and everything bar ElasticSearch is singing sweetly again. On start, we get an IndexFormatTooNewException. Not many get that one, but CheckIndex appears to be worth a go, except that also barfs: nicks@elasticsearch:/var/lib/elasticsearch/elasticsearch/nodes/0/indices/support/3/index$ sudo java -cp /usr/share/elasticsearch/lib/lucene-core-4.8.1.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex . -fix Opening index @ . ERROR: could not read any segments file in directory org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource: BufferedChecksumIndexInput(MMapIndexInput(path=/var/lib/elasticsearch/elasticsearch/nodes/0/indices/support/3/index/segments.gen))): 1071082519 (needs to be between -2 and -3) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:803) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:710) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:412) at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393) at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1967) This has rendered our ES service offline. Any suggestions to fix it would be appreciated, we are uncertain how to proceed otherwise. ES 1.2.1 from Debian packages. We are just (now) upgrading to 1.4 to see if this results in some progress. James -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMH6%2BaxYVvHEnjLocbeHh9v5pjReZ0aDds0aOtkjy1sDGmfY1w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Recover data from corrupted index
Hy,
I have a corrupted index in an elasticsearch cluster. The index is
corrupted due to bad mappings. As you can see in example, there are two
fields with same name and different mappings - one is string/doc_values and
one is date, and in Lucene this is not possible, due to doc_values
settings.
{
test1: {
properties: {
date: {
format: dd//: HH: mm: ssZ,
type: date
}
}
}
}
{
test2: {
properties: {
date: {
index: not_analyzed,
fielddata: {
format: doc_values
},
doc_values: true
}
}
}
}
The problem is that I'm not able to start the index in order to reindex the
data from it. (The error is [WARN ][cluster.action.shard ] [cluster]
[index][1] sending failed shard for [index][1],
node[b2yMUmvXQFy8LFd8ei6DZQ], [P], s[INITIALIZING], indexUUID
[I_zoR-4SS02WEfRNKFc6MA], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[index][1] failed recovery]; nested:
IllegalArgumentException[cannot change DocValues type from BINARY to
SORTED_SET for field date]; ]]).
Is it possible to somehow recover the data from corrupted index and put it
in a healthy index ? (I'm using Elasticsearch 1.1.1)
Thank you,
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8d3f0501-9fb8-4406-ae82-b85f7f29185a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
