[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2795: --------------------------------------- Attachment: LUCENE-2795.patch Hi Varun! Thanks for checking back here... I ran this directory on Linux and hit a number of test failures, from two bugs in the directory impl. First, NativeUnixIndexInput.fileLength() failed to include bytes still in the buffer; second, on clone, if the current II was positioned at EOF and EOF was on a (512 byte) page boundary then we'd hit a false IOE. I fixed those... I also changed NativeUnixDir to take a delegate Dir which we fall back to when we don't want to use direct IO; this enabled removing of open_normal since we just use the "normal" java Dir impls for this. To run all tests on Linux I do this: {noformat} export CLASSPATH=/lucene/unixdir/lucene/build/contrib/misc/lucene-misc-4.0-SNAPSHOT.jar export LD_LIBRARY_PATH=/lucene/unixdir/lucene/build/native:/usr/local/lib ant test-core -Dtests.directory=NativeUnixDirectory {noformat} (Change the path to your full path... or maybe use relative path but that could be dangerous since tests may change the CWD... not sure). It also requires a temporary ctor in the directory taking only File and using FSDirectory.open for the delegate directory. Finally I added another setting, minBytesDirect: if the file to open (in openInput), or expected merge size (in createOutput), is smaller than this, then we don't use direct IO. To test this... I temporarily fixed the dir to always use direct IO, and tests (eventually: ~75 minutes!) passed, except for some Solr tests which look like timeout problems. I temporarily dropped the default buffer size to 4 KB (from 256 KB) else the JVM ran out of direct buffer space. I'll test on OS X as well... > Genericize DirectIOLinuxDir -> UnixDir > -------------------------------------- > > Key: LUCENE-2795 > URL: https://issues.apache.org/jira/browse/LUCENE-2795 > Project: Lucene - Java > Issue Type: Improvement > Components: core/store > Reporter: Michael McCandless > Assignee: Varun Thacker > Labels: gsoc2011, lucene-gsoc-11, mentor > Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, > LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, > LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch > > > Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to > use it for indexWriter and not IndexReader (searching). It's a trap. > But, once we do LUCENE-2793, we can make it fully general purpose because > then a single native Dir impl can be used. > I'd also like to make it generic to other Unices, if we can, so that it > becomes UnixDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org