Re: Solr 4.0, slow opening searchers
In addition to Alan's comment, are you doing any warmup queries? Your Solr logs should show you some interesting stats, and the admin page also has some stats about warmup times. Although I'd expect similar issues when reopening searchers if it was just warmup queries. But 267M docs on a single machine (spread over 9 cores or not) is quite a lot (depending, of course, on how beefy the machine is and the characteristics of your corpus). It's possible you're just I/O bound at startup, experiencing memory pressure, etc. that is, your index is just too large for your hardware. I've seen machines vary from 10M to 300M docs being reasonable. FWIW, Erick On Fri, Jan 11, 2013 at 12:31 PM, Alan Woodward a...@flax.co.uk wrote: Hi Marcel, Are you committing data with hard commits or soft commits? I've seen systems where we've inadvertently only used soft commits, which means that the entire transaction log has to be re-read on startup, which can take a long time. Hard commits flush indexed data to disk, and make it a lot quicker to restart. Alan Woodward a...@flax.co.uk On 11 Jan 2013, at 13:51, Marcel Bremer wrote: Hi, We're experiencing slow startup times of searchers in Solr when containing a large number of documents. We use Solr v4.0 with Jetty and currently have 267.657.634 documents stored, spread across 9 cores. These documents contain keywords, with additional statistics, which we are using for suggestions and related keywords. When we (re)start Solr on one of our servers it can take up to two hours before Solr has opened all of it's searchers and starts accepting connections again. We can't figure out why it takes so long to open those searchers. Also the CPU and memory usage of Solr while opening searchers is not extremely high. Are there any known issues or tips someone could give us to speed up opening searchers? If you need more details, please ping me. Best regards, Marcel Bremer Vinden.nl BV
Solr 4.0, slow opening searchers
Hi, We're experiencing slow startup times of searchers in Solr when containing a large number of documents. We use Solr v4.0 with Jetty and currently have 267.657.634 documents stored, spread across 9 cores. These documents contain keywords, with additional statistics, which we are using for suggestions and related keywords. When we (re)start Solr on one of our servers it can take up to two hours before Solr has opened all of it's searchers and starts accepting connections again. We can't figure out why it takes so long to open those searchers. Also the CPU and memory usage of Solr while opening searchers is not extremely high. Are there any known issues or tips someone could give us to speed up opening searchers? If you need more details, please ping me. Best regards, Marcel Bremer Vinden.nl BV
Re: Solr 4.0, slow opening searchers
Hi Marcel, Are you committing data with hard commits or soft commits? I've seen systems where we've inadvertently only used soft commits, which means that the entire transaction log has to be re-read on startup, which can take a long time. Hard commits flush indexed data to disk, and make it a lot quicker to restart. Alan Woodward a...@flax.co.uk On 11 Jan 2013, at 13:51, Marcel Bremer wrote: Hi, We're experiencing slow startup times of searchers in Solr when containing a large number of documents. We use Solr v4.0 with Jetty and currently have 267.657.634 documents stored, spread across 9 cores. These documents contain keywords, with additional statistics, which we are using for suggestions and related keywords. When we (re)start Solr on one of our servers it can take up to two hours before Solr has opened all of it's searchers and starts accepting connections again. We can't figure out why it takes so long to open those searchers. Also the CPU and memory usage of Solr while opening searchers is not extremely high. Are there any known issues or tips someone could give us to speed up opening searchers? If you need more details, please ping me. Best regards, Marcel Bremer Vinden.nl BV
