Yes the index can fit in ram on the boxes I am testing with - Its the main rationale for sharding to make sure that we can hold an index in ram at all times.
MADV_WILLNEED might be rather bad if the index is bigger than ram (something to test maybe) On 12/11/2012 02:40 AM, Michael McCandless wrote: > Yes, do keep digging! darklaunch sounds like a very useful tool :) > And big bonus points because it's written in Python, heh. > > The results are incredible. > > In this test is there plenty of RAM to hold the index? > > Mike McCandless > > http://blog.mikemccandless.com > > On Tue, Dec 11, 2012 at 12:33 AM, Greg Bowyer <[email protected]> wrote: >> Since its too long (and has too much HTML and pictures and such forth) >> for the mailing list I have a more detailed write up here >> >> http://people.apache.org/~gbowyer/madvise-perf/index.html >> >> However the short version. >> >> At $DAYJOB as part of moving to lucene 4.0 I have been looking at what >> we can change in our modes of thinking, on of these relates (for us) to >> having a separate legacy system that serves the stored data for the >> search engine (which we imaginatively call docserve). >> >> Whilst playing with this I noticed that I lost a lot of performance >> rather quickly, after a lot of digging it looks like getting a call in >> mmaping to madvise (specifically a call of the form madvise(addr, >> MADV_WILLNEED) might improve search performance. >> >> Anyone have any thoughts and or ideas here, I am going to try to get a >> whole heap more investigation done before I start messing with lucene's >> behavior on mmap (because whilst it improves /my/ performance a lot it >> might be I only noticed it since I am poking an open-wound YMMV), but >> there is something of interest here. >> >> Then again this might be my own personal insanity .. :S >> >> -- Greg >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
