Invalid handle?
Hi all, I use multi-threads to do indexing work, sometimes I get invalid handle exception. I am certain that I use the hit results before closing the reader. What's wrong with me? Pls help me! -- Thanks, Jiang
Re: setNorm/commit issue on Windows
: If you have an index that has separate norms (.sN or .fN files), and : you open a reader against this index, and then try to use another : reader to change (& commit) the norms, that second reader will hit an : IOException (Access Denied) on Windows during commit because the first setNorms is definitely some funky voodoo -- not intended for general use. It would nto suprise me in the least if using it while another IndexReader was opened didn't work -- on Windows or any other platform. Given it's nature, i would treat the results of using it during concurrent access from other threads/process the same as IndexReader.unlock -- scary, and unadvised. -Hoss - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-665) temporary file access denied on Windows
[ http://issues.apache.org/jira/browse/LUCENE-665?page=comments#action_12430919 ] Doron Cohen commented on LUCENE-665: > just to confirm, is it the COMMIT lock that's throwing these > unhandled exceptions (not the WRITE lock)? > If so, lockless commits would fix this. In my tests so far, these errors appeared only for commit locks. However I consider this a coincidence - there is nothing as far as I can understand special with commit locks comparing to write locks - in particular they both use createNewFile. So, I agree that lockless commits would prevent this, which is good, but we cannot count on that it would not happen for write locks as well. Also, the more I think about it the more I like lock-less commits, still, they would take a while to get into Lucene, while this simple fix can help easily now. Last, with lock-less commits, still, there would be calls to createNewFile for write lock, and there would be calls to renameFile() and other IO file operations, intensively. By having a safety code like the retry logic that is invoked only in rare cases of these unexpected, some nasty errors would be reduced, more users would be happy. > Can you provide more details on the exceptions you're seeing? > Especially on the "cannot rename file" exception? Here is one from my run log, that occurs at the call to optimize, after at the end of all the add-remove iterations - [junit] java.io.IOException: Cannot rename C:\Documents and Settings\tpowner\Local Settings\Temp\test.perf\index_24\deleteable.new to C:\Documents and Settings\tpowner\Local Settings\Temp\test.perf\index_24\deletable [junit] at org.apache.lucene.store.FSDirectory.doRenameFile(FSDirectory.java:328) [junit] at org.apache.lucene.store.FSDirectory.renameFile(FSDirectory.java:280) [junit] at org.apache.lucene.index.IndexWriter.writeDeleteableFiles(IndexWriter.java:967) [junit] at org.apache.lucene.index.IndexWriter.deleteSegments(IndexWriter.java:911) [junit] at org.apache.lucene.index.IndexWriter.commitChanges(IndexWriter.java:872) [junit] at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:823) [junit] at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:798) [junit] at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:614) [junit] at org.apache.lucene.index.IndexModifier.optimize(IndexModifier.java:304) [junit] at org.apache.lucene.index.TestBufferedDeletesPerf.doOptimize(TestBufferedDeletesPerf.java:266) [junit] at org.apache.lucene.index.TestBufferedDeletesPerf.measureInterleavedAddRemove(TestBufferedDeletesPerf.java:218) [junit] at org.apache.lucene.index.TestBufferedDeletesPerf.doTestBufferedDeletesPerf(TestBufferedDeletesPerf.java:144) [junit] at org.apache.lucene.index.TestBufferedDeletesPerf.testBufferedDeletesPerfCase7(TestBufferedDeletesPerf.java:134) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [junit] at java.lang.reflect.Method.invoke(Method.java:585) [junit] at junit.framework.TestCase.runTest(TestCase.java:154) [junit] at junit.framework.TestCase.runBare(TestCase.java:127) [junit] at junit.framework.TestResult$1.protect(TestResult.java:106) [junit] at junit.framework.TestResult.runProtected(TestResult.java:124) [junit] at junit.framework.TestResult.run(TestResult.java:109) [junit] at junit.framework.TestCase.run(TestCase.java:118) [junit] at junit.framework.TestSuite.runTest(TestSuite.java:208) [junit] at junit.framework.TestSuite.run(TestSuite.java:203) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:297) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:672) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:567) [junit] Caused by: java.io.FileNotFoundException: C:\Documents and Settings\tpowner\Local Settings\Temp\test.perf\index_24\deletable (Access is denied) [junit] at java.io.FileOutputStream.open(Native Method) [junit] at java.io.FileOutputStream.(FileOutputStream.java:179) [junit] at java.io.FileOutputStream.(FileOutputStream.java:131) [junit] at org.apache.lucene.store.FSDirectory.doRenameFile(FSDirectory.java:312) [junit] ... 27 more This exception btw is from the performance test for interleaved-adds-and-removes - issue 565 - so IndexWriter line numbers here relate to applying recent patch from issue 565 (though the same errors are obtained with the
setNorm/commit issue on Windows
I'm working on the lock-less commit patch, and came across what seems to be a bug in the current Lucene sources: If you have an index that has separate norms (.sN or .fN files), and you open a reader against this index, and then try to use another reader to change (& commit) the norms, that second reader will hit an IOException (Access Denied) on Windows during commit because the first reader is holding open the .sN (or .fN) files. The .del files don't have this issue because they opened, read, and then closed. Has anyone seen this? Maybe the IndexReader.setNorm() API is not typically used when there are other active searchers (on Windows)? Does this sound like a real bug (ie this use case is supposed to work)? I couldn't find anything matching this in Jira. Mike - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: java.io.IOException: Access is denied on java.io.WinNTFileSystem.createFileExclusively
Transplanting this thread off of java-user to discuss a more java-dev question I have (below): http://www.gossamer-threads.com/lists/lucene/java-user/39382 Yonik Seeley wrote: On 8/27/06, Doron Cohen <[EMAIL PROTECTED]> wrote: I plan to submit an update to that patch later today accommodating your comments (and others); It will most probably retry for IOExceptions (not analyzing the exception text); also, it would return false if the *retry* for obtain() failed with exception. Lock.obtain(timeout) already has retry/timeout logic around Lock.obtain(), so wouldn't the change to return false on an IOException be sufficient? User level applications should then use obtain(timeout) rather than obtain() unless they want to do timeout/retry logic themselves. I would agree: the retry/timeout logic already exists above so let's just return false on hitting any IOException when calling createNewFile. (Doron, this still wouldn't fix the other strange "across the board" IO problems you see on Windows. I'd really like to reproduce this but so far (running your stress test) I can't -- are you sure it's not a virus checker or something?). I think on hitting the IOException on java.io.File.createNewFile, we should return false (so timeout/retry logic can handle it). However we should also message this to the user to make debugging simple (vs just silently swallowing the exception). But: how to do so? I like the infoStream that IndexWriter supports, but, should we adopt [one of] the Java logging APIs (Apache commons logging, java.util.logging, log4j, etc.)? As far as I can tell Lucene doesn't do so now, so, what's the "right way" to let the user know that an IOException occurred and this is why the "lock obtain" failed? Mike - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-635) [PATCH] Decouple locking implementation from Directory implementation
[ http://issues.apache.org/jira/browse/LUCENE-635?page=all ] Michael McCandless updated LUCENE-635: -- Attachment: LUCENE-635-Aug27.patch OK, I agree. I've updated the CHANGES.txt to state this small change. And I've fixed SimpleFSLockFactory to move directory existence checking & creation back into the obtain() method. New patch attached! > [PATCH] Decouple locking implementation from Directory implementation > - > > Key: LUCENE-635 > URL: http://issues.apache.org/jira/browse/LUCENE-635 > Project: Lucene - Java > Issue Type: Improvement > Components: Index >Affects Versions: 2.0.0 >Reporter: Michael McCandless >Priority: Minor > Attachments: LUCENE-635-Aug27.patch, LUCENE-635-Aug3.patch, > patch-Jul26.tar > > > This is a spinoff of http://issues.apache.org/jira/browse/LUCENE-305. > I've opened this new issue to capture that it's wider scope than > LUCENE-305. > This is a patch originally created by Jeff Patterson (see above link) > and then modified as described here: > http://issues.apache.org/jira/browse/LUCENE-305#action_12418493 > with some small additional changes: > * For each FSDirectory.getDirectory(), I made a corresponding > version that also accepts a LockFactory instance. So, you can > construct an FSDirectory with your own LockFactory. > * Cascaded defaulting for FSDirectory's LockFactory implementation: > if you pass in a LockFactory instance, it's used; else if > setDisableLocks was called, we use NoLockFactory; else, if the > system property "org.apache.lucene.store.FSDirectoryLockFactoryClass" > is defined, we use that; finally, we'll use the original locking > implementation (SimpleFSLockFactory). > The gist is that all locking code has been moved out of *Directory and > into subclasses of a new abstract LockFactory class. You can now set > the LockFactory of a Directory to change how it does locking. For > example, you can create an FSDirectory but set its locking to > SingleInstanceLockFactory (if you know all writing/reading will take > place a single JVM). > The changes pass all unit tests (on Ubuntu Linux Sun Java 1.5 and > Windows XP Sun Java 1.4), and I added another TestCase to test the > LockFactory code. > Note that LockFactory defaults are not changed: FSDirectory defaults > to SimpleFSLockFactory and RAMDirectory defaults to > SingleInstanceLockFactory. > Next step (separate issue) is to create a LockFactory that uses the OS > native locks (through java.nio). -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-664) [PATCH] small fixes to the new scoring.html doc
[ http://issues.apache.org/jira/browse/LUCENE-664?page=all ] Michael McCandless updated LUCENE-664: -- Attachment: scoring-small-fixes3.patch Doh! You are correct. OK I re-based the diffs off the current version. Sorry about that. > [PATCH] small fixes to the new scoring.html doc > --- > > Key: LUCENE-664 > URL: http://issues.apache.org/jira/browse/LUCENE-664 > Project: Lucene - Java > Issue Type: Improvement > Components: Website >Affects Versions: 2.0.1 >Reporter: Michael McCandless > Attachments: lucene.uxf, scoring-small-fixes.patch, > scoring-small-fixes2.patch, scoring-small-fixes3.patch > > > This is an awesome initiative. We need more docs that cleanly explain the > inner workings of Lucene in general... thanks Grant & Steve & others! > I have a few small initial proposed fixes, largely just adding some more > description around the components of the formula. But also a couple typos, > another link out to Wikipedia, a missing closing ), etc. I've only made it > through the "Understanding the Scoring Formula" section so far. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
