London OSS search social - meetup 6th June

2012-06-02 Thread Richard Marr
Apologies for the short notice guys, we're meeting up at The Plough in Bloomsbury on Wednesday 6th June. As usual the format is open and there's a healthy mix of experience and backgrounds. Please come and share wisdom, ask questions, geek out, etc. in the presence of beverages. -- Richard Marr

Re: OOM during IndexReader open

2012-06-02 Thread Michael McCandless
It could be your index has an unusual number of unique terms. If you can upgrade to the latest 3.x, the RAM used by the terms index has been very substantially reduced... If not, try setting the termInfosIndexDivisor to eg 2 or 3 ... this will load 1/2 or 1/3 of the indexed terms into RAM, but ma

Re: OOM during IndexReader open

2012-06-02 Thread Jack Krupansky
Have you tried a System.gc before opening the index? And check how much memory is available before the open. Does the open seem to be taking more memory that could reasonably be expected? How big is the index, how many docs, how many unique terms in that index? 1.8G doesn't sound like a lot. Y

Re: Deferring merging of index segments

2012-06-02 Thread Michael McCandless
On Fri, Jun 1, 2012 at 8:09 PM, Vitaly Funstein wrote: > Yes, I am only calling IndexWriter.addDocument() OK. > Interestingly, relative performance of either approach seems to greatly > depend on the number of documents per index. In both types of runs, I used > 10 writer threads, each writing d