Salman, Warming up may be useful if your caches are getting decent hit ratios. Plus, you are warming up the OS cache when you warm up.
Otis ---- Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ ----- Original Message ---- > From: Salman Akram <salman.ak...@northbaysolutions.net> > To: solr-user@lucene.apache.org > Sent: Fri, February 4, 2011 3:33:41 PM > Subject: Re: Performance optimization of Proximity/Wildcard searches > > I know so we are not really using it for regular warm-ups (in any case index > is updated on hourly basis). Just tried few times to compare results. The > issue is I am not even sure if warming up is useful for such regular > updates. > > > > On Fri, Feb 4, 2011 at 5:16 PM, Otis Gospodnetic <otis_gospodne...@yahoo.com > > wrote: > > > Salman, > > > > I only skimmed your email, but wanted to say that this part sounds a little > > suspicious: > > > > > Our warm up script currently executes all distinct queries in our logs > > > having count > 5. It was run yesterday (with all the indexing update > > every > > > > It sounds like this will make warmup take a looooong time, assuming you > > have > > more than a handful distinct queries in your logs. > > > > Otis > > ---- > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch > > Lucene ecosystem search :: http://search-lucene.com/ > > > > > > > > ----- Original Message ---- > > > From: Salman Akram <salman.ak...@northbaysolutions.net> > > > To: solr-user@lucene.apache.org; t...@statsbiblioteket.dk > > > Sent: Tue, January 25, 2011 6:32:48 AM > > > Subject: Re: Performance optimization of Proximity/Wildcard searches > > > > > > By warmed index you only mean warming the SOLR cache or OS cache? As I > > said > > > our index is updated every hour so I am not sure how much SOLR cache > > would > > > be helpful but OS cache should still be helpful, right? > > > > > > I haven't compared the results with a proper script but from manual > > testing > > > here are some of the observations. > > > > > > 'Recent' queries which are in cache of course return immediately (only > > if > > > they are exactly same - even if they took 3-4 mins first time). I will > > need > > > to test how many recent queries stay in cache but still this would work > > only > > > for very common queries. User can run different queries and I want at > > least > > > them to be at 'acceptable' level (5-10 secs) even if not very fast. > > > > > > Our warm up script currently executes all distinct queries in our logs > > > having count > 5. It was run yesterday (with all the indexing update > > every > > > hour after that) and today when I executed some of the same queries > > again > > > their time seemed a little less (around 15-20%), I am not sure if this > > means > > > anything. However, still their time is not acceptable. > > > > > > What do you think is the best way to compare results? First run all the > > warm > > > up queries and then execute same randomly and compare? > > > > > > We are using Windows server, would it make a big difference if we move > > to > > > Linux? Our load is not high but some queries are really complex. > > > > > > Also I was hoping to move to SSD in last after trying out all software > > > options. Is that an agreed fact that on large indexes (which don't fit > > in > > > RAM) proximity/wildcard/phrase queries (on common words) would be slow > > and > > > it can be only improved by cache warm up and better hardware? Otherwise > > with > > > an index of around 150GB such queries will take more than a min? > > > > > > If that's the case I know this question is very subjective but if a > > single > > > query takes 2 min on SAS 10K RPM what would its approx time be on a good > > SSD > > > (everything else same)? > > > > > > Thanks! > > > > > > > > > On Tue, Jan 25, 2011 at 3:44 PM, Toke Eskildsen > > <t...@statsbiblioteket.dk>wrote: > > > > > > > On Tue, 2011-01-25 at 10:20 +0100, Salman Akram wrote: > > > > > Cache warming is a good option too but the index get updated every > > hour > > > > so > > > > > not sure how much would that help. > > > > > > > > What is the time difference between queries with a warmed index and a > > > > cold one? If the warmed index performs satisfactory, then one answer > > is > > > > to upgrade your underlying storage. As always for IO-caused > > performance > > > > problem in Lucene/Solr-land, SSD is the answer. > > > > > > > > > > > > > > > > > -- > > > Regards, > > > > > > Salman Akram > > > > > > > > > -- > Regards, > > Salman Akram >
