I'm seeing an odd error during indexing for which I can't find any reason.
The relevant solr log entry:
2017-03-24 19:09:35.363 ERROR (commitScheduler-30-thread-1) [
x:build0324] o.a.s.u.CommitTracker auto commit
error...:java.io.EOFException: read past EOF:
MMapIndexInput(path="/indexes/solrindexes/build0324/index/_4ku.fdx")
at
org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75)
...
Suppressed: org.apache.lucene.index.CorruptIndexException: checksum
status indeterminate: remaining=0, please run checkindex for more details
(resource=
BufferedChecksumIndexInput(MMapIndexInput(path="/indexes/solrindexes/build0324/index/_4ku.fdx")))
at
org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:451)
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:140)
followed within a few seconds by
2017-03-24 19:09:56.402 ERROR (commitScheduler-31-thread-1) [
x:build0324] o.a.s.u.CommitTracker auto commit
error...:org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1820)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1931)
...
Caused by: java.io.EOFException: read past EOF:
MMapIndexInput(path="/indexes/solrindexes/build0324/index/_4ku.fdx")
at
org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:75)
This error is repeated a few times as the indexing continued and further
autocommits were triggered.
I stopped the indexing process, made a backup snapshot of the index,
restarted indexing at a checkpoint, and everything then completed without
further incidents
I ran checkIndex on the saved snapshot and it reported no errors
whatsoever. Operations on the complete index (inclcuing an optimize and
several query scripts) have all been error-free.
Some background:
Solr information from the beginning of the checkindex output:
-------
Opening index @ /indexes/solrindexes/build0324.bad/index
Segments file=segments_9s numSegments=105 version=6.3.0
id=7m1ldieoje0m6sljp7xocbz9l userData={commitTimeMSec=1490400514324}
1 of 105: name=_be maxDoc=1227144
version=6.3.0
id=7m1ldieoje0m6sljp7xocburb
codec=Lucene62
compound=false
numFiles=14
size (MB)=4,926.186
diagnostics = {os=Linux, java.vendor=Oracle Corporation,
java.version=1.8.0_45, java.vm.version=25.45-b02, lucene.version=6.3.0,
mergeMaxNumSegments=-1, os.arch=amd64, java.runtime.version=1.8.0_45-b13,
source=merge, mergeFactor=19, os.version=3.10.0-229.1.2.el7.x86_64,
timestamp=1490380905920}
no deletions
test: open reader.........OK [took 0.176 sec]
test: check integrity.....OK [took 37.399 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [49 fields] [took 0.000 sec]
test: field norms.........OK [17 fields] [took 0.030 sec]
test: terms, freq, prox...OK [14568108 terms; 612537186 terms/docs
pairs; 801208966 tokens] [took 30.005 sec]
test: stored fields.......OK [150164874 total field count; avg 122.4
fields per doc] [took 35.321 sec]
test: term vectors........OK [4804967 total term vector count; avg 3.9
term/freq vector fields per doc] [took 55.857 sec]
test: docvalues...........OK [4 docvalues fields; 0 BINARY; 1 NUMERIC;
2 SORTED; 0 SORTED_NUMERIC; 1 SORTED_SET] [took 0.954 sec]
test: points..............OK [0 fields, 0 points] [took 0.000 sec]
-----
The indexing process is a Python script (using the scorched Python
client) which spawns multiple instance of itself, in this case 6, so there
are definitely concurrent calls ( to /update/json )
Solrconfig and the schema have not been changed for several months, during
which time many ingests have been done, and the documents which were being
indexed at the time of the error have been indexed before without problems,
so I don't think it's a data issue.
I saw the same error occur earlier in the day, and decided at that time to
delete the core and restart the Solr instance.
The server is an Amazon instance running CentOS 7. I checked the system
logs and didn't see any evidence of hardware errors
I'm puzzled as to why this would start happening out of the blue and I
can't find any partiuclarly relevant posts to this forum or Stackexchange.
Anyone have an idea what's going on ?
-Simon
