[
https://issues.apache.org/jira/browse/LUCENE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133804#comment-14133804
]
Sanne Grinovero commented on LUCENE-5541:
-----------------------------------------
Thanks Michael,
[~gustavonalle] from the Infinispan team was able to reproduce it and figure
out how this relates to {{File.exists}};
https://issues.jboss.org/browse/ISPN-2981 is now being resolved, it seems this
wasn't a bug in Lucene but a very subtle race condition in the Infinispan
Directory for Lucene, so affecting the Directory for Lucene 4 as well.
For the record ISPN-2981 would only trigger if all following conditions are met:
- A Merge is writing concurrently to a thread using an IndexWriter for doing
new writes
- The node in the cluster happens to not be the primary owner for a specific
entry (so it would be impossible on single node tests, unlikely on small
clusters)
- High load (or rather: low write load would make it unlikely)
> FileExistsCachingDirectory, to work around unreliable File.exists
> -----------------------------------------------------------------
>
> Key: LUCENE-5541
> URL: https://issues.apache.org/jira/browse/LUCENE-5541
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/store
> Reporter: Michael McCandless
> Attachments: LUCENE-5541.patch
>
>
> File.exists is a dangerous method in Java, because if there is a
> low-level IOException (permission denied, out of file handles, etc.)
> the method can return false when it should return true.
> Fortunately, as of Lucene 4.x, we rely much less on File.exists,
> because we track which files the codec components created, and we know
> those files then exist.
> But, unfortunately, going from 3.0.x to 3.6.x, we increased our
> reliance on File.exists, e.g. when creating CFS we check File.exists
> on each sub-file before trying to add it, and I have a customer
> corruption case where apparently a transient low level IOE caused
> File.exists to incorrectly return false for one of the sub-files. It
> results in corruption like this:
> {noformat}
> java.io.FileNotFoundException: No sub-file with id .fnm found
> (fileName=_1u7.cfs files: [.tis, .tii, .frq, .prx, .fdt, .nrm, .fdx])
>
> org.apache.lucene.index.CompoundFileReader.openInput(CompoundFileReader.java:157)
>
> org.apache.lucene.index.CompoundFileReader.openInput(CompoundFileReader.java:146)
> org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> org.apache.lucene.index.IndexWriter.getFieldInfos(IndexWriter.java:1212)
>
> org.apache.lucene.index.IndexWriter.getCurrentFieldInfos(IndexWriter.java:1228)
> org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1161)
> {noformat}
> I think typically local file systems don't often hit such low level
> errors, but if you have an index on a remote filesystem, where network
> hiccups can cause problems, it's more likely.
> As a simple workaround, I created a basic Directory delegator that
> holds a Set of all created but not deleted files, and short-circuits
> fileExists to return true if the file is in that set.
> I don't plan to commit this: we aren't doing bug-fix releases on
> 3.6.x anymore (it's very old by now), and this problem is already
> "fixed" in 4.x (by reducing our reliance on File.exists), but I wanted
> to post the code here in case others hit it. It looks like it was hit
> e.g. https://netbeans.org/bugzilla/show_bug.cgi?id=189571 and
> https://issues.jboss.org/browse/ISPN-2981
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
