[ http://issues.apache.org/jira/browse/LUCENE-665?page=all ]
Doron Cohen updated LUCENE-665: ------------------------------- Attachment: FSDirs_Retry_Logic_3.patch I am attaching an updated patch - FSDirs_Retry_Logic_3.patch. In this update: - merge with code changes by issue 635 ("decouple locking from directory") - modified by recommendations in above comments: - do not rely on specific exception message text. - overide lock.obtain(timeout) and handle unexpected exceptions there. - do not modify logic of obtain() (no changes to this method). - UNEXPECTED_ERROR_RETRY_DELAY set to 100ms. - debug prints commented out. "ant test" tests all pass. My stress IO test passes as well. > temporary file access denied on Windows > --------------------------------------- > > Key: LUCENE-665 > URL: http://issues.apache.org/jira/browse/LUCENE-665 > Project: Lucene - Java > Issue Type: Bug > Components: Store > Affects Versions: 2.0.0 > Environment: Windows > Reporter: Doron Cohen > Attachments: FSDirectory_Retry_Logic.patch, > FSDirs_Retry_Logic_3.patch, Test_Output.txt, TestInterleavedAddAndRemoves.java > > > When interleaving adds and removes there is frequent opening/closing of > readers and writers. > I tried to measure performance in such a scenario (for issue 565), but the > performance test failed - the indexing process crashed consistently with > file "access denied" errors - "cannot create a lock file" in > "lockFile.createNewFile()" and "cannot rename file". > This is related to: > - issue 516 (a closed issue: "TestFSDirectory fails on Windows") - > http://issues.apache.org/jira/browse/LUCENE-516 > - user list questions due to file errors: > - > http://www.nabble.com/OutOfMemory-and-IOException-Access-Denied-errors-tf1649795.html > - > http://www.nabble.com/running-a-lucene-indexing-app-as-a-windows-service-on-xp%2C-crashing-tf2053536.html > - discussion on lock-less commits > http://www.nabble.com/Lock-less-commits-tf2126935.html > My test setup is: XP (SP1), JAVA 1.5 - both SUN and IBM SDKs. > I noticed that the problem is more frequent when locks are created on one > disk and the index on another. Both are NTFS with Windows indexing service > enabled. I suspect this indexing service might be related - keeping files > busy for a while, but don't know for sure. > After experimenting with it I conclude that these problems - at least in my > scenario - are due to a temporary situation - the FS, or the OS, is > *temporarily* holding references to files or folders, preventing from > renaming them, deleting them, or creating new files in certain directories. > So I added to FSDirectory a retry logic in cases the error was related to > "Access Denied". This is the same approach brought in > http://www.nabble.com/running-a-lucene-indexing-app-as-a-windows-service-on-xp%2C-crashing-tf2053536.html > - there, in addition to the retry, gc() is invoked (I did not gc()). This is > based on the *hope* that a access-denied situation would vanish after a small > delay, and the retry would succeed. > I modified FSDirectory this way for "Access Denied" errors during creating a > new files, renaming a file. > This worked fine for me. The performance test that failed before, now managed > to complete. There should be no performance implications due to this > modification, because only the cases that would otherwise wrongly fail are > now delaying some extra millis and retry. > I am attaching here a patch - FSDirectory_Retry_Logic.patch - that has these > changes to FSDirectory. > All "ant test" tests pass with this patch. > Also attaching a test case that demostrates the problem - at least on my > machine. There two tests cases in that test file - one that works in system > temp (like most Lucene tests) and one that creates the index in a different > disk. The latter case can only run if the path ("D:" , "tmp") is valid. > It would be great if people that experienced these problems could try out > this patch and comment whether it made any difference for them. > If it turns out useful for others as well, including this patch in the code > might help to relieve some of those "frustration" user cases. > A comment on state of proposed patch: > - It is not a "ready to deploy" code - it has some debug printing, showing > the cases that the "retry logic" actually took place. > - I am not sure if current 30ms is the right delay... why not 50ms? 10ms? > This is currently defined by a constant. > - Should a call to gc() be added? (I think not.) > - Should the retry be attempted also on "non access-denied" exceptions? (I > think not). > - I feel it is somewhat "woodoo programming", but though I don't like it, it > seems to work... > Attached files: > 1. TestInterleavedAddAndRemoves.java - the LONG test that fails on XP without > the patch and passes with the patch. > 2. FSDirectory_Retry_Logic.patch > 3. Test_Output.txt- output of the test with the patch, on my XP. Only the > createNewFile() case had to be bypassed in this test, but for another program > I also saw the renameFile() being bypassed. > - Doron -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]