[jira] [Updated] (LUCENE-6835) Directory.deleteFile should "own" retrying deletions on Windows
[ https://issues.apache.org/jira/browse/LUCENE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-6835: --- Fix Version/s: 6.0 > Directory.deleteFile should "own" retrying deletions on Windows > --- > > Key: LUCENE-6835 > URL: https://issues.apache.org/jira/browse/LUCENE-6835 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Michael McCandless > Fix For: 5.5, master, 6.0 > > Attachments: LUCENE-6835.patch, LUCENE-6835.patch > > > Rob's idea: > Today, we have hairy logic in IndexFileDeleter to deal with Windows file > systems that cannot delete still open files. > And with LUCENE-6829, where OfflineSorter now must deal with the situation > too ... I worked around it by fixing all tests to disable the virus checker. > I think it makes more sense to push this "platform specific problem" lower in > the stack, into Directory? I.e., its deleteFile method would catch the > access denied, and then retry the deletion later. Then we could re-enable > virus checker on all these tests, simplify IndexFileDeleter, etc. > Maybe in the future we could further push this down, into WindowsDirectory, > and fix FSDirectory.open to return WindowsDirectory on windows ... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6835) Directory.deleteFile should "own" retrying deletions on Windows
[
https://issues.apache.org/jira/browse/LUCENE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-6835:
---
Attachment: LUCENE-6835.patch
I think the branch is in pretty good shape ... tests pass, even when forcing
{{VirusCheckingFS}} usage.
I'm attaching an applyable a "createPatch.py" patch.
I have one nocommit left: what to do about IndexFileDeleter's previous heroics
around not removing any files referenced by a stale segments_N when that
segments_N itself cannot be deleted. I think we should just remove this
(OS-specific) behavior?
I'm a little nervous about the changes to FSDirectory, holding onto pending
deletions, retrying them periodically on new write operations ... but I do
think it's important we get this OS-specific behavior in Lucene down lower in
the stack ...
I also fixed the new {{VirusCheckingFS}} to not use {{Random}} instance
anymore, but do its own pseudo-random behavior based on the hash of the
incoming {{Path}}.
> Directory.deleteFile should "own" retrying deletions on Windows
> ---
>
> Key: LUCENE-6835
> URL: https://issues.apache.org/jira/browse/LUCENE-6835
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Michael McCandless
> Fix For: 5.5, master
>
> Attachments: LUCENE-6835.patch, LUCENE-6835.patch
>
>
> Rob's idea:
> Today, we have hairy logic in IndexFileDeleter to deal with Windows file
> systems that cannot delete still open files.
> And with LUCENE-6829, where OfflineSorter now must deal with the situation
> too ... I worked around it by fixing all tests to disable the virus checker.
> I think it makes more sense to push this "platform specific problem" lower in
> the stack, into Directory? I.e., its deleteFile method would catch the
> access denied, and then retry the deletion later. Then we could re-enable
> virus checker on all these tests, simplify IndexFileDeleter, etc.
> Maybe in the future we could further push this down, into WindowsDirectory,
> and fix FSDirectory.open to return WindowsDirectory on windows ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6835) Directory.deleteFile should "own" retrying deletions on Windows
[ https://issues.apache.org/jira/browse/LUCENE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-6835: -- Fix Version/s: (was: 5.4) 5.5 > Directory.deleteFile should "own" retrying deletions on Windows > --- > > Key: LUCENE-6835 > URL: https://issues.apache.org/jira/browse/LUCENE-6835 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Michael McCandless > Fix For: 5.5, Trunk > > Attachments: LUCENE-6835.patch > > > Rob's idea: > Today, we have hairy logic in IndexFileDeleter to deal with Windows file > systems that cannot delete still open files. > And with LUCENE-6829, where OfflineSorter now must deal with the situation > too ... I worked around it by fixing all tests to disable the virus checker. > I think it makes more sense to push this "platform specific problem" lower in > the stack, into Directory? I.e., its deleteFile method would catch the > access denied, and then retry the deletion later. Then we could re-enable > virus checker on all these tests, simplify IndexFileDeleter, etc. > Maybe in the future we could further push this down, into WindowsDirectory, > and fix FSDirectory.open to return WindowsDirectory on windows ... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6835) Directory.deleteFile should "own" retrying deletions on Windows
[ https://issues.apache.org/jira/browse/LUCENE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-6835: --- Attachment: LUCENE-6835.patch Tentative initial patch, but there are problems. I removed IFD's hairy code about retrying deletes (yay!), and moved it down into FSDirectory, so it now becomes a Directory impl's job to "deal with" finicky filesystems that prevent deletion of open files. I also changed Directory.deleteFile to Directory.deleteFiles, so "revisit pending deletions" is less O(N^2). The big problem with the patch now is I completely disabled MDW's virus/open-file checker, because we are no longer allowed to "fake" a virus checker in a Directory wrapper (since it's now Directory's job to do retries on deletes) ... I think to move forward w/ this approach we must also re-implement the virus checker "down low" inside Mock FS Some tests are still angry because they rely on MDW's not deleting still open files ... > Directory.deleteFile should "own" retrying deletions on Windows > --- > > Key: LUCENE-6835 > URL: https://issues.apache.org/jira/browse/LUCENE-6835 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Michael McCandless > Fix For: Trunk, 5.4 > > Attachments: LUCENE-6835.patch > > > Rob's idea: > Today, we have hairy logic in IndexFileDeleter to deal with Windows file > systems that cannot delete still open files. > And with LUCENE-6829, where OfflineSorter now must deal with the situation > too ... I worked around it by fixing all tests to disable the virus checker. > I think it makes more sense to push this "platform specific problem" lower in > the stack, into Directory? I.e., its deleteFile method would catch the > access denied, and then retry the deletion later. Then we could re-enable > virus checker on all these tests, simplify IndexFileDeleter, etc. > Maybe in the future we could further push this down, into WindowsDirectory, > and fix FSDirectory.open to return WindowsDirectory on windows ... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
