Michael McCandless created LUCENE-7113:
------------------------------------------
Summary: OfflineSorter and BKD should verify checksums in their
temp files
Key: LUCENE-7113
URL: https://issues.apache.org/jira/browse/LUCENE-7113
Project: Lucene - Core
Issue Type: Improvement
Reporter: Michael McCandless
Assignee: Michael McCandless
Fix For: master, 6.0
I am trying to index all 3.2 B points from the latest OpenStreetMaps export.
My SSDs were not up to this, so I added a spinning magnets disk to beast2.
But then I was hitting scary bug-like exceptions
({{ArrayIndexOutOfBoundsException}}) when indexing the first 2B points, and I
finally checked dmesg and saw that my hard drive is dying.
I think it's important that our temp file usages also validate checksums (like
we do for all our index files, either at reader open or at merge or
{{CheckIndex}}), so we can hopefully easily differentiate a bit-flipping IO
system from a possible Lucene bug, in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
