[
http://issues.apache.org/jira/browse/LUCENE-665?page=comments#action_12434509 ]
Michael McCandless commented on LUCENE-665:
-------------------------------------------
I just sent this summary of this to java-user:
There is an issue opened on Lucene:
http://issues.apache.org/jira/browse/LUCENE-665
that I'd like to draw your attention to and summarize here because
recently users have hit it.
The gist of the issue is: on Windows, you sometimes see intermittant
"Access Denied" errors in renaming segments.new to segments or
deletable.new to deletable, etc. Lucene typically writes files first
to X.new and then renames then to X.
I know there was at least one recent thread where someone was hitting
this and there have been others in the past (including other Jira
issues).
Anyway, at the end of the issue it was discovered that there was an
unrelated piece of software (TortoiseSVN client) installed which was
using a filesystem "change log" capability in Windows that was
"causing" the problem: uninstalling it made the errors go away.
Unfortunately, there are apparently many software packages that use
this "change log" capability in Windows (virus checkers, Microsoft's
indexing service, etc.) and so the above issue remains open to figure
out whether / how to make Lucene robust to these cases.
But the bottom line is: if you hit these "Access Denied" errors, one
workaround is to try to turn off or uninstall the software that might
be doing this. I realize in many cases that's not an option (it's a
production box; you can't turn off virus checkers; etc.), but at least
it's something to try if you can, until there's some resolution on
that issue.
> temporary file access denied on Windows
> ---------------------------------------
>
> Key: LUCENE-665
> URL: http://issues.apache.org/jira/browse/LUCENE-665
> Project: Lucene - Java
> Issue Type: Bug
> Components: Store
> Affects Versions: 2.0.0
> Environment: Windows
> Reporter: Doron Cohen
> Attachments: FSDirectory_Retry_Logic.patch,
> FSDirs_Retry_Logic_3.patch, Test_Output.txt, TestInterleavedAddAndRemoves.java
>
>
> When interleaving adds and removes there is frequent opening/closing of
> readers and writers.
> I tried to measure performance in such a scenario (for issue 565), but the
> performance test failed - the indexing process crashed consistently with
> file "access denied" errors - "cannot create a lock file" in
> "lockFile.createNewFile()" and "cannot rename file".
> This is related to:
> - issue 516 (a closed issue: "TestFSDirectory fails on Windows") -
> http://issues.apache.org/jira/browse/LUCENE-516
> - user list questions due to file errors:
> -
> http://www.nabble.com/OutOfMemory-and-IOException-Access-Denied-errors-tf1649795.html
> -
> http://www.nabble.com/running-a-lucene-indexing-app-as-a-windows-service-on-xp%2C-crashing-tf2053536.html
> - discussion on lock-less commits
> http://www.nabble.com/Lock-less-commits-tf2126935.html
> My test setup is: XP (SP1), JAVA 1.5 - both SUN and IBM SDKs.
> I noticed that the problem is more frequent when locks are created on one
> disk and the index on another. Both are NTFS with Windows indexing service
> enabled. I suspect this indexing service might be related - keeping files
> busy for a while, but don't know for sure.
> After experimenting with it I conclude that these problems - at least in my
> scenario - are due to a temporary situation - the FS, or the OS, is
> *temporarily* holding references to files or folders, preventing from
> renaming them, deleting them, or creating new files in certain directories.
> So I added to FSDirectory a retry logic in cases the error was related to
> "Access Denied". This is the same approach brought in
> http://www.nabble.com/running-a-lucene-indexing-app-as-a-windows-service-on-xp%2C-crashing-tf2053536.html
> - there, in addition to the retry, gc() is invoked (I did not gc()). This is
> based on the *hope* that a access-denied situation would vanish after a small
> delay, and the retry would succeed.
> I modified FSDirectory this way for "Access Denied" errors during creating a
> new files, renaming a file.
> This worked fine for me. The performance test that failed before, now managed
> to complete. There should be no performance implications due to this
> modification, because only the cases that would otherwise wrongly fail are
> now delaying some extra millis and retry.
> I am attaching here a patch - FSDirectory_Retry_Logic.patch - that has these
> changes to FSDirectory.
> All "ant test" tests pass with this patch.
> Also attaching a test case that demostrates the problem - at least on my
> machine. There two tests cases in that test file - one that works in system
> temp (like most Lucene tests) and one that creates the index in a different
> disk. The latter case can only run if the path ("D:" , "tmp") is valid.
> It would be great if people that experienced these problems could try out
> this patch and comment whether it made any difference for them.
> If it turns out useful for others as well, including this patch in the code
> might help to relieve some of those "frustration" user cases.
> A comment on state of proposed patch:
> - It is not a "ready to deploy" code - it has some debug printing, showing
> the cases that the "retry logic" actually took place.
> - I am not sure if current 30ms is the right delay... why not 50ms? 10ms?
> This is currently defined by a constant.
> - Should a call to gc() be added? (I think not.)
> - Should the retry be attempted also on "non access-denied" exceptions? (I
> think not).
> - I feel it is somewhat "woodoo programming", but though I don't like it, it
> seems to work...
> Attached files:
> 1. TestInterleavedAddAndRemoves.java - the LONG test that fails on XP without
> the patch and passes with the patch.
> 2. FSDirectory_Retry_Logic.patch
> 3. Test_Output.txt- output of the test with the patch, on my XP. Only the
> createNewFile() case had to be bypassed in this test, but for another program
> I also saw the renameFile() being bypassed.
> - Doron
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
