[jira] [Updated] (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[
https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-2795:
---
Attachment: LUCENE-2795.patch
Hi Varun!
Thanks for checking back here...
I ran this directory on Linux and hit a number of test failures, from
two bugs in the directory impl. First,
NativeUnixIndexInput.fileLength() failed to include bytes still in the
buffer; second, on clone, if the current II was positioned at EOF
and EOF was on a (512 byte) page boundary then we'd hit
a false IOE. I fixed those...
I also changed NativeUnixDir to take a delegate Dir which we fall back
to when we don't want to use direct IO; this enabled removing of
open_normal since we just use the "normal" java Dir impls for this.
To run all tests on Linux I do this:
{noformat}
export
CLASSPATH=/lucene/unixdir/lucene/build/contrib/misc/lucene-misc-4.0-SNAPSHOT.jar
export LD_LIBRARY_PATH=/lucene/unixdir/lucene/build/native:/usr/local/lib
ant test-core -Dtests.directory=NativeUnixDirectory
{noformat}
(Change the path to your full path... or maybe use relative path but
that could be dangerous since tests may change the CWD... not sure).
It also requires a temporary ctor in the directory taking only File
and using FSDirectory.open for the delegate directory.
Finally I added another setting, minBytesDirect: if the file to open
(in openInput), or expected merge size (in createOutput), is smaller
than this, then we don't use direct IO.
To test this... I temporarily fixed the dir to always use direct IO,
and tests (eventually: ~75 minutes!) passed, except for some Solr tests which
look
like timeout problems. I temporarily dropped the default
buffer size to 4 KB (from 256 KB) else the JVM ran out of direct
buffer space.
I'll test on OS X as well...
> Genericize DirectIOLinuxDir -> UnixDir
> --
>
> Key: LUCENE-2795
> URL: https://issues.apache.org/jira/browse/LUCENE-2795
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/store
>Reporter: Michael McCandless
>Assignee: Varun Thacker
> Labels: gsoc2011, lucene-gsoc-11, mentor
> Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch,
> LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch,
> LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch
>
>
> Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to
> use it for indexWriter and not IndexReader (searching). It's a trap.
> But, once we do LUCENE-2793, we can make it fully general purpose because
> then a single native Dir impl can be used.
> I'd also like to make it generic to other Unices, if we can, so that it
> becomes UnixDirectory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[
https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated LUCENE-2795:
--
Attachment: LUCENE-2795.patch
I updated the patch so that it applies cleanly with the current trunk revision.
{noformat}ant build-native-unix{noformat} Comiles successfully on a 64 bit
Ubuntu Linux machine running kernel version 3.0.0-14-generic.
{noformat}ant-DDirectory=NativeUnixDirectory test-core{noformat}
Runs successfully.
Is more testing required before committing it?
> Genericize DirectIOLinuxDir -> UnixDir
> --
>
> Key: LUCENE-2795
> URL: https://issues.apache.org/jira/browse/LUCENE-2795
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/store
>Reporter: Michael McCandless
>Assignee: Varun Thacker
> Labels: gsoc2011, lucene-gsoc-11, mentor
> Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch,
> LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch,
> LUCENE-2795.patch, LUCENE-2795.patch
>
>
> Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to
> use it for indexWriter and not IndexReader (searching). It's a trap.
> But, once we do LUCENE-2793, we can make it fully general purpose because
> then a single native Dir impl can be used.
> I'd also like to make it generic to other Unices, if we can, so that it
> becomes UnixDirectory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated LUCENE-2795: -- Attachment: LUCENE-2795.patch Updated patch. http://ant.apache.org/manual/Tasks/conditions.html#os With this I can get information on the kernel. But the problem is "Linux" is not a separate family attribute. Any pointers on how to go about it so that we can bq. Also, we should run some more tests here, on modern kernels, since the SEQUENTIAL fadvise flag is better implemented in kernels >= 2.6.29; see this comment: bq. http://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html?showComment=0#c2572106601600642254 bq. It could be, if kernel is new enough, we should not do DIRECT and instead do SEQUENTIAL. > Genericize DirectIOLinuxDir -> UnixDir > -- > > Key: LUCENE-2795 > URL: https://issues.apache.org/jira/browse/LUCENE-2795 > Project: Lucene - Java > Issue Type: Improvement > Components: core/store >Reporter: Michael McCandless >Assignee: Varun Thacker > Labels: gsoc2011, lucene-gsoc-11, mentor > Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, > LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, > LUCENE-2795.patch > > > Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to > use it for indexWriter and not IndexReader (searching). It's a trap. > But, once we do LUCENE-2793, we can make it fully general purpose because > then a single native Dir impl can be used. > I'd also like to make it generic to other Unices, if we can, so that it > becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated LUCENE-2795: -- Attachment: LUCENE-2795.patch Sorry for the delay. I have made the changes suggested. > Genericize DirectIOLinuxDir -> UnixDir > -- > > Key: LUCENE-2795 > URL: https://issues.apache.org/jira/browse/LUCENE-2795 > Project: Lucene - Java > Issue Type: Improvement > Components: core/store >Reporter: Michael McCandless >Assignee: Varun Thacker > Labels: gsoc2011, lucene-gsoc-11, mentor > Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, > LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch > > > Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to > use it for indexWriter and not IndexReader (searching). It's a trap. > But, once we do LUCENE-2793, we can make it fully general purpose because > then a single native Dir impl can be used. > I'd also like to make it generic to other Unices, if we can, so that it > becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[
https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated LUCENE-2795:
--
Attachment: LUCENE-2795.patch
Updated jdocs.
createOutput switches between direct and normal.
Added constructors NativeUnixIndexOutput/NativeUnixIndexInput which does not
take buffersize and uses 256kb by default.
Changed logic of using forcedbuffersize as suggested - It is only used in MERGE
context rest of the times BII.buffersize
I am still trying to figure out how to find out different kernel versions in
this case. This is what I found out till now. Use the system("uname -r") call
in the c program directly. Use a makefile. Still haven't figured out how to do
so using cpptasks though.
> Genericize DirectIOLinuxDir -> UnixDir
> --
>
> Key: LUCENE-2795
> URL: https://issues.apache.org/jira/browse/LUCENE-2795
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/store
>Reporter: Michael McCandless
>Assignee: Varun Thacker
> Labels: gsoc2011, lucene-gsoc-11, mentor
> Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch,
> LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch
>
>
> Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to
> use it for indexWriter and not IndexReader (searching). It's a trap.
> But, once we do LUCENE-2793, we can make it fully general purpose because
> then a single native Dir impl can be used.
> I'd also like to make it generic to other Unices, if we can, so that it
> becomes UnixDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer updated LUCENE-2795: Attachment: LUCENE-2795.patch here is an updated patch that works on OSX. I had to copy the old DirectIOLinuxDirectory.java to NativeUnixDirectory.java (svn copy) to apply the patch cleanly. I added an additional path to the JDKs header files and a linker argument otherwise I got a linker error. GCC by default doesn't pass -lstdc++ to the linker > Genericize DirectIOLinuxDir -> UnixDir > -- > > Key: LUCENE-2795 > URL: https://issues.apache.org/jira/browse/LUCENE-2795 > Project: Lucene - Java > Issue Type: Improvement > Components: core/store >Reporter: Michael McCandless >Assignee: Varun Thacker > Labels: gsoc2011, lucene-gsoc-11, mentor > Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, > LUCENE-2795.patch, LUCENE-2795.patch > > > Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to > use it for indexWriter and not IndexReader (searching). It's a trap. > But, once we do LUCENE-2793, we can make it fully general purpose because > then a single native Dir impl can be used. > I'd also like to make it generic to other Unices, if we can, so that it > becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated LUCENE-2795: -- Attachment: LUCENE-2795.patch Changed DirectIOLinuxDirectory to NativeUnixDirectory and also made some other changes. > Genericize DirectIOLinuxDir -> UnixDir > -- > > Key: LUCENE-2795 > URL: https://issues.apache.org/jira/browse/LUCENE-2795 > Project: Lucene - Java > Issue Type: Improvement > Components: core/store >Reporter: Michael McCandless >Assignee: Varun Thacker > Labels: gsoc2011, lucene-gsoc-11, mentor > Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, > LUCENE-2795.patch > > > Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to > use it for indexWriter and not IndexReader (searching). It's a trap. > But, once we do LUCENE-2793, we can make it fully general purpose because > then a single native Dir impl can be used. > I'd also like to make it generic to other Unices, if we can, so that it > becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated LUCENE-2795: -- Attachment: LUCENE-2795.patch I'm not sure this is the correct/best way to this. > Genericize DirectIOLinuxDir -> UnixDir > -- > > Key: LUCENE-2795 > URL: https://issues.apache.org/jira/browse/LUCENE-2795 > Project: Lucene - Java > Issue Type: Improvement > Components: core/store >Reporter: Michael McCandless >Assignee: Varun Thacker > Labels: gsoc2011, lucene-gsoc-11, mentor > Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch > > > Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to > use it for indexWriter and not IndexReader (searching). It's a trap. > But, once we do LUCENE-2793, we can make it fully general purpose because > then a single native Dir impl can be used. > I'd also like to make it generic to other Unices, if we can, so that it > becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated LUCENE-2795: -- Attachment: LUCENE-2795.patch the open_direct, posix_fadvise function in onlylinux.h remains the same. In onlybsd.h open_direct, posix_fadvise are the same too except that the O_NOATIME flag is not present. In onlyosx the open_direct is implemented in a different way. Also I have added a open_normal function to all of the headers which will be used in case the IOContext in not a MERGE. > Genericize DirectIOLinuxDir -> UnixDir > -- > > Key: LUCENE-2795 > URL: https://issues.apache.org/jira/browse/LUCENE-2795 > Project: Lucene - Java > Issue Type: Improvement > Components: core/store >Reporter: Michael McCandless >Assignee: Varun Thacker > Labels: gsoc2011, lucene-gsoc-11, mentor > Attachments: LUCENE-2795.patch, LUCENE-2795.patch > > > Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to > use it for indexWriter and not IndexReader (searching). It's a trap. > But, once we do LUCENE-2793, we can make it fully general purpose because > then a single native Dir impl can be used. > I'd also like to make it generic to other Unices, if we can, so that it > becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated LUCENE-2795: -- Attachment: LUCENE-2795.patch Initial Patch.. This will apply to the IOContext branch. This patch also contains some uncommitted code from LUCENE-2793. In NativePosixUtil.cpp the open_direct method where I use fcntl() for OS X could not be compiled on my machine because of the OS specific F_NOCACHE flag used. Let me know if the usage is correct. > Genericize DirectIOLinuxDir -> UnixDir > -- > > Key: LUCENE-2795 > URL: https://issues.apache.org/jira/browse/LUCENE-2795 > Project: Lucene - Java > Issue Type: Improvement > Components: core/store >Reporter: Michael McCandless >Assignee: Simon Willnauer > Labels: gsoc2011, lucene-gsoc-11, mentor > Attachments: LUCENE-2795.patch > > > Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to > use it for indexWriter and not IndexReader (searching). It's a trap. > But, once we do LUCENE-2793, we can make it fully general purpose because > then a single native Dir impl can be used. > I'd also like to make it generic to other Unices, if we can, so that it > becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer updated LUCENE-2795: Labels: gsoc2011 lucene-gsoc-11 mentor (was: gsoc2011, lucene-gsoc-11 mentor,) > Genericize DirectIOLinuxDir -> UnixDir > -- > > Key: LUCENE-2795 > URL: https://issues.apache.org/jira/browse/LUCENE-2795 > Project: Lucene - Java > Issue Type: Improvement > Components: Store >Reporter: Michael McCandless > Labels: gsoc2011, lucene-gsoc-11, mentor > > Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to > use it for indexWriter and not IndexReader (searching). It's a trap. > But, once we do LUCENE-2793, we can make it fully general purpose because > then a single native Dir impl can be used. > I'd also like to make it generic to other Unices, if we can, so that it > becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer updated LUCENE-2795: Labels: gsoc2011, lucene-gsoc-11 mentor, (was: mentor) > Genericize DirectIOLinuxDir -> UnixDir > -- > > Key: LUCENE-2795 > URL: https://issues.apache.org/jira/browse/LUCENE-2795 > Project: Lucene - Java > Issue Type: Improvement > Components: Store >Reporter: Michael McCandless > Labels: gsoc2011,, lucene-gsoc-11, mentor, > > Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to > use it for indexWriter and not IndexReader (searching). It's a trap. > But, once we do LUCENE-2793, we can make it fully general purpose because > then a single native Dir impl can be used. > I'd also like to make it generic to other Unices, if we can, so that it > becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer updated LUCENE-2795: Labels: mentor (was: gsoc2011 lucene-gsoc-11) > Genericize DirectIOLinuxDir -> UnixDir > -- > > Key: LUCENE-2795 > URL: https://issues.apache.org/jira/browse/LUCENE-2795 > Project: Lucene - Java > Issue Type: Improvement > Components: Store >Reporter: Michael McCandless > Labels: mentor > > Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to > use it for indexWriter and not IndexReader (searching). It's a trap. > But, once we do LUCENE-2793, we can make it fully general purpose because > then a single native Dir impl can be used. > I'd also like to make it generic to other Unices, if we can, so that it > becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-2795) Genericize DirectIOLinuxDir -> UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-2795: --- Labels: gsoc2011 lucene-gsoc-11 (was: ) > Genericize DirectIOLinuxDir -> UnixDir > -- > > Key: LUCENE-2795 > URL: https://issues.apache.org/jira/browse/LUCENE-2795 > Project: Lucene - Java > Issue Type: Improvement > Components: Store >Reporter: Michael McCandless > Labels: gsoc2011, lucene-gsoc-11 > > Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to > use it for indexWriter and not IndexReader (searching). It's a trap. > But, once we do LUCENE-2793, we can make it fully general purpose because > then a single native Dir impl can be used. > I'd also like to make it generic to other Unices, if we can, so that it > becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
