Also, what was stated is that "this is not a safe assumption" is
ludicrous. It happens to not be safe because X software package that
was installed is holding locks on files.... If could be that Y
software package (virus scanner !) deletes files automatically when
it doesn't recognize an extension - or possibly one of the lucene
index files ends up matching some virus signature.
You always must be aware of the environment in which you install
anyway software. Having indexing software or virus scanning software
run on the directories that contains Lucene files is just a bad
(incorrect) deployment - will perform badly as best, and not at all
at worst.
On Sep 13, 2006, at 2:52 PM, Bruce Ritchie wrote:
While what you say is true about indexing should be disabled, that
really doesn't solve the actual issue. Administrators of applications
using lucene often do not have control over the actual machine and
thus
cannot determine what is and is not installed. Besides that, many
of us
do development on Windows machines and don't want the hassle of being
forced to run the application either remotely on a unix box or in a VM
just to work around this issue. I've hit this exact issue during
development - it's truly an annoying issue that a simple sleep/retry
loop should resolve.
Regards,
Bruce Ritchie
-----Original Message-----
From: robert engels [mailto:[EMAIL PROTECTED]
Sent: Wednesday, September 13, 2006 3:41 PM
To: java-dev@lucene.apache.org
Subject: Re: [jira] Commented: (LUCENE-665) temporary file access
denied
on Windows
This is a server application. Windows indexing service should be
disable
on the directories that contain Lucene locks and files.
This is the same procedure that would be required for any database.
On Sep 13, 2006, at 2:27 PM, Michael McCandless (JIRA) wrote:
[ http://issues.apache.org/jira/browse/LUCENE-665?
page=comments#action_12434527 ]
Michael McCandless commented on LUCENE-665:
-------------------------------------------
I do think we should make Lucene robust to "windows change log"
software.
We could take the position that you have to uninstall such software
because they "conflict" with Lucene, but I don't think that's
realistic. Apparently many packages use this convenient API and that
will only get worse with time.
I would put this under the "Lucene should assume the least common
denominator of filesystem's capabilities" umbrella. Meaning, Lucene
now assumes it can rename files right after closing them, but on
Windows this isn't a safe assumption so if possible we should change
the index format to not require this.
I will try to reproduce this bug with my [upcoming] changes for
lockless commits (numbered segments files) -- the lockless commits
changes do much less file renaming, so the issue should be rarer (but
could still occur).
temporary file access denied on Windows
---------------------------------------
Key: LUCENE-665
URL: http://issues.apache.org/jira/browse/LUCENE-665
Project: Lucene - Java
Issue Type: Bug
Components: Store
Affects Versions: 2.0.0
Environment: Windows
Reporter: Doron Cohen
Attachments: FSDirectory_Retry_Logic.patch,
FSDirs_Retry_Logic_3.patch, Test_Output.txt,
TestInterleavedAddAndRemoves.java
When interleaving adds and removes there is frequent opening/
closing
of readers and writers.
I tried to measure performance in such a scenario (for issue 565),
but the performance test failed - the indexing process crashed
consistently with file "access denied" errors - "cannot create a
lock
file" in "lockFile.createNewFile()" and "cannot rename file".
This is related to:
- issue 516 (a closed issue: "TestFSDirectory fails on Windows") -
http://issues.apache.org/jira/browse/LUCENE-516
- user list questions due to file errors:
- http://www.nabble.com/OutOfMemory-and-IOException-Access-
Denied-errors-tf1649795.html
- http://www.nabble.com/running-a-lucene-indexing-app-as-a-
windows-service-on-xp%2C-crashing-tf2053536.html
- discussion on lock-less commits http://www.nabble.com/Lock-less-
commits-tf2126935.html My test setup is: XP (SP1), JAVA 1.5 - both
SUN and IBM SDKs.
I noticed that the problem is more frequent when locks are
created on
one disk and the index on another. Both are NTFS with Windows
indexing service enabled. I suspect this indexing service might be
related - keeping files busy for a while, but don't know for sure.
After experimenting with it I conclude that these problems - at
least
in my scenario - are due to a temporary situation - the FS, or the
OS, is *temporarily* holding references to files or folders,
preventing from renaming them, deleting them, or creating new files
in certain directories.
So I added to FSDirectory a retry logic in cases the error was
related to "Access Denied". This is the same approach brought in
http://www.nabble.com/running-a-lucene-indexing-app-as-a-windows-
service-on-xp%2C-crashing-tf2053536.html - there, in addition to the
retry, gc() is invoked (I did not gc()). This is based on the
*hope* that a access-denied situation would vanish after a small
delay, and the retry would succeed.
I modified FSDirectory this way for "Access Denied" errors during
creating a new files, renaming a file.
This worked fine for me. The performance test that failed before,
now
managed to complete. There should be no performance implications due
to this modification, because only the cases that would otherwise
wrongly fail are now delaying some extra millis and retry.
I am attaching here a patch - FSDirectory_Retry_Logic.patch - that
has these changes to FSDirectory.
All "ant test" tests pass with this patch.
Also attaching a test case that demostrates the problem - at
least on
my machine. There two tests cases in that test file - one that works
in system temp (like most Lucene tests) and one that creates the
index in a different disk. The latter case can only run if the path
("D:" , "tmp") is valid.
It would be great if people that experienced these problems could
try
out this patch and comment whether it made any difference for them.
If it turns out useful for others as well, including this patch in
the code might help to relieve some of those "frustration" user
cases.
A comment on state of proposed patch:
- It is not a "ready to deploy" code - it has some debug printing,
showing the cases that the "retry logic" actually took place.
- I am not sure if current 30ms is the right delay... why not 50ms?
10ms? This is currently defined by a constant.
- Should a call to gc() be added? (I think not.)
- Should the retry be attempted also on "non access-denied"
exceptions? (I think not).
- I feel it is somewhat "woodoo programming", but though I don't
like
it, it seems to work...
Attached files:
1. TestInterleavedAddAndRemoves.java - the LONG test that fails
on XP
without the patch and passes with the patch.
2. FSDirectory_Retry_Logic.patch
3. Test_Output.txt- output of the test with the patch, on my XP.
Only the createNewFile() case had to be bypassed in this test, but
for another program I also saw the renameFile() being bypassed.
- Doron
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the
administrators: http://issues.apache.org/jira/secure/
Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/
software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
