RE: Lucene going non-responsive under heavy load

2009-12-23 Thread Uwe Schindler
iraj Haider [mailto:si...@jobdiva.com] > Sent: Wednesday, December 23, 2009 12:00 AM > To: java-user@lucene.apache.org > Subject: Re: Lucene going non-responsive under heavy load > > Hi Uwe, > Following are the info: > Java VM: "1.6.0_05" > Java Options: -server -Xms512m

Re: Lucene going non-responsive under heavy load

2009-12-23 Thread Michael McCandless
Are you using IndexReader.reopen to open a new reader, from an existing one? That's much more efficient than opening a new reader. I think a good next step is to run with IndexWriter.setInfoStream on, and run your JRE with verbose GC, to see more details. Mike On Wed, Dec 23, 2009 at 9:12 AM, S

Re: Lucene going non-responsive under heavy load

2009-12-23 Thread Siraj Haider
We have dual cpu intel xeon machines running "Red Hat Enterprise Linux ES release 3 (Taroon Update 6)". We have 4GB memory on these machines with 2GB allocated to tomcat. After modifying the index we open a new one, warm it up, make it live and then close the old one. -siraj Michael McCandle

Re: Lucene going non-responsive under heavy load

2009-12-22 Thread Michael McCandless
A merge shouldn't make the machine completely non-responsive, just, slower to run searches / index documents. What kind of machine / IO system is this? You can set maxMergeDocs to limit how large the merge is allowed to be. But, be careful, since if you set this too small you'll wind up with way

Re: Lucene going non-responsive under heavy load

2009-12-22 Thread Siraj Haider
Hi Uwe, Following are the info: Java VM: "1.6.0_05" Java Options: -server -Xms512m -Xmx2048m We are not using any specific GC. regards -siraj Uwe Schindler wrote: If it suddenly gets unresponsive, you may have a GC (garbage collector) problem. Can you post your Java options, Java VM version, m

Re: Lucene going non-responsive under heavy load

2009-12-22 Thread Siraj Haider
Hi Mike, You are right, sometimes there is an implicit merge running when the machine goes non-responsive. How can we avoid running those merges during the day and how can we minimize the effect it will have on searches? -siraj Michael McCandless wrote: Is it possible a large merge is runni

RE: Lucene going non-responsive under heavy load

2009-12-22 Thread Uwe Schindler
If it suddenly gets unresponsive, you may have a GC (garbage collector) problem. Can you post your Java options, Java VM version, max heap and so on. Maybe you are doing some strange things. For us the best GC to use (we have Java 1.5) was the "Parallel New GC" / "The Concurrent Low Pause Collecto

Re: Lucene going non-responsive under heavy load

2009-12-22 Thread Michael McCandless
Is it possible a large merge is running? You can turn on IndexWriter.setInfoStream to see more details about what IW is doing, including merging. Mike On Tue, Dec 22, 2009 at 5:19 PM, Siraj Haider wrote: > Hello guys, > We have a dilemma on a few of our lucene machines.  We have a tomcat runnin