Hi Varun, Those two issues would make a great GSoC! Comments below...
On Tue, Apr 5, 2011 at 1:56 PM, Varun Thacker <varunthacker1...@gmail.com> wrote: > I would like to combine two tasks as part of my project > namely-Directory createOutput and openInput should take an IOContext > (Lucene-2793) and compliment it by Generalize DirectIOLinuxDir to > UnixDir (Lucene-2795). > > The first part of the project is aimed at significantly reducing time > taken to search during indexing by adding an IOContext which would > store buffer size and have options to bypass the OS’s buffer cache > (This is what causes the slowdown in search ) and other hints. Once > completed I would move on to Lucene-2795 and generalize the Directory > implementation to make a UnixDirectory . So, the first part (LUCENE-2793) should cause no change at all to performance, functionality, etc., because it's "merely" installing the plumbing (IOContext threaded throughout the low-level store APIs in Lucene) so that higher levels can send important details down to the Directory. We'd fix IndexWriter/IndexReader to fill out this IOContext with the details (merging, flushing, new reader, etc.). There's some fun/freedom here in figuring out just what details should be included in IOContext... (eg: is it low level "set buffer size to 4 KB" or is it high level "I am opening a new near-real-time reader"). This first step is a rote cutover, just changing APIs but in no way taking advantage of the new APIs. The 2nd step (LUCENE-2795) would then take advantage of this plumbing, by creating a UnixDir impl that, using JNI (C code), passes advanced flags when opening files, based on the incoming IOContext. The goal is a single UnixDir that has ifdefs so that it's usable across multiple Unices, and eg would use direct IO if the context is merging. If we are ambitious we could rope Windows into the mix, too, and then this would be NativeDir... We can measure success by validating that a big merge while searching does not hurt search performance? (Ie we should be able to reproduce the results from http://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html). > I have spoken to Micheal McCandless and Simon Willnauer about > undertaking these tasks. Micheal McCandless has agreed to mentor me . > I would love to be able to contribute and learn from Apache Lucene > community this summer. Also I would love suggestions on how to make my > application proposal stronger. I think either Simon or I can be the "official" mentor, and then the other one of us (and other Lucene committers) will support/chime in... This is an important change for Lucene! Mike --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
