Thanks once again for the helpful suggestions! Regarding the selection of facet fields, I think publishDate (which is actually just a year) and callnumber-first (which is actually a very broad, high-level category) are okay. authorStr is an interesting problem: it's definitely a useful facet (when a user searches for an author, odds are good that they want the one who published the most books... i.e. a search for dickens will probably show Charles Dickens at the top of the facet list), but it has a long tail since there are many minor authors who have only published one or two books... Is there a possibility that the facet.mincount parameter could be helpful here, or does that have no impact on performance/memory footprint?
Regarding polling interval for slaves, are you referring to a distributed Solr environment, or is this something to do with Solr's internals? We're currently a single-server environment, so I don't think I have to worry if it's related to a multi-server setup... but if it's something internal, could you point me to the right area of the admin panel to check my stats? I'm not seeing anything about polling on the statistics page. It's also a little strange that all of my warmupTime stats on searchers and caches are showing as 0 -- is that normal? thanks, Demian > -----Original Message----- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Friday, June 03, 2011 4:45 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr performance tuning - disk i/o? > > Quick impressions: > > The faceting is usually best done on fields that don't have lots of > unique > values for three reasons: > 1> It's questionable how much use to the user to have a gazillion > facets. > In the case of a unique field per document, in fact, it's useless. > 2> resource requirements go up as a function of the number of unique > terms. This is true for faceting and sorting. > 3> warmup times grow the more terms have to be read into memory. > > > Glancing at your warmup stuff, things like publishDate, authorStr and > maybe > callnumber-first are questionable. publishDate depends on how coarse > the > resolution is. If it's by day, that's not really much use. authorStr.. > How many > authors have more than one publication? Would this be better served by > some > kind of autosuggest rather than facets? callnumber-first... I don't > really know, but > if it's unique per document it's probably not something the user would > find useful > as a facet. > > The admin page will help you determine the number of unique terms per > field, > which may guide you whether or not to continue to facet on these > fields. > > As Otis said, doing a sort on the fields during warmup will also help. > > Watch your polling interval for any slaves in relation to the warmup > times. > If your polling interval is shorter than the warmup times, you run a > risk of > "runaway warmups". > > As you've figured out, measuring responses to the first few queries > doesn't > always measure what you really need <G>.. > > I don't have the pages handy, but autowarming is a good topic to > understand, > so you might spend some time tracking it down. > > Best > Erick > > On Fri, Jun 3, 2011 at 11:21 AM, Demian Katz > <demian.k...@villanova.edu> wrote: > > Thanks to you and Otis for the suggestions! Some more information: > > > > - Based on the Solr stats page, my caches seem to be working pretty > well (few or no evictions, hit rates in the 75-80% range). > > - VuFind is actually doing two Solr queries per search (one initial > search followed by a supplemental spell check search -- I believe this > is necessary because VuFind has two separate spelling indexes, one for > shingled terms and one for single words). That is probably > exaggerating the problem, though based on searches with debugQuery on, > it looks like it's always the initial search (rather than the > supplemental spelling search) that's consuming the bulk of the time. > > - enableLazyFieldLoading is set to true. > > - I'm retrieving 20 documents per page. > > - My JVM settings: -server - > Xloggc:/usr/local/vufind/solr/jetty/logs/gc.log -Xms4096m -Xmx4096m - > XX:+UseParallelGC -XX:+UseParallelOldGC -XX:NewRatio=5 > > > > It appears that a large portion of my problem had to do with > autowarming, a topic that I've never had a strong grasp on, though > perhaps I'm finally learning (any recommended primer links would be > welcome!). I did have some autowarming settings in solrconfig.xml (an > arbitrary search for a bunch of random keywords in the newSearcher and > firstSearcher events, plus autowarmCount settings on all of my caches). > However, when I looked at the debugQuery output, I noticed that a huge > amount of time was being wasted loading facets on the first search > after restarting Solr, so I changed my newSearcher and firstSearcher > events to this: > > > > <arr name="queries"> > > <lst> > > <str name="q">*:*</str> > > <str name="start">0</str> > > <str name="rows">10</str> > > <str name="facet">true</str> > > <str name="facet.mincount">1</str> > > <str name="facet.field">collection</str> > > <str name="facet.field">format</str> > > <str name="facet.field">publishDate</str> > > <str name="facet.field">callnumber-first</str> > > <str name="facet.field">topic_facet</str> > > <str name="facet.field">authorStr</str> > > <str name="facet.field">language</str> > > <str name="facet.field">genre_facet</str> > > <str name="facet.field">era_facet</str> > > <str name="facet.field">geographic_facet</str> > > </lst> > > </arr> > > > > Overall performance has now increased dramatically, and now the > biggest bottleneck in the debug output seems to be the shingle spell > checking! > > > > Any other suggestions are welcome, since I suspect there's still room > to squeeze more performance out of the system, and I'm still not sure > I'm making the most of autowarming... but this seems like a big step > in the right direction. Thanks again for the help! > > > > - Demian > > > >> -----Original Message----- > >> From: Erick Erickson [mailto:erickerick...@gmail.com] > >> Sent: Friday, June 03, 2011 9:41 AM > >> To: solr-user@lucene.apache.org > >> Subject: Re: Solr performance tuning - disk i/o? > >> > >> This doesn't seem right. Here's a couple of things to try: > >> 1> attach &debugQuery=on to your long-running queries. The QTime > >> returned > >> is the time taken to search, NOT including the time to load the > >> docs. That'll > >> help pinpoint whether the problem is the search itself, or > >> assembling the > >> documents. > >> 2> Are you autowarming? If so, be sure it's actually done before > >> querying. > >> 3> Measure queries after the first few, particularly if you're > sorting > >> or > >> faceting. > >> 4> What are your JVM settings? How much memory do you have? > >> 5> is <enableLazyFieldLoading> set to true in your solrconfig.xml? > >> 6> How many docs are you returning? > >> > >> > >> There's more, but that'll do for a start.... Let us know if you > gather > >> more data > >> and it's still slow. > >> > >> Best > >> Erick > >> > >> On Fri, Jun 3, 2011 at 8:44 AM, Demian Katz > <demian.k...@villanova.edu> > >> wrote: > >> > Hello, > >> > > >> > I'm trying to move a VuFind installation from an ailing physical > >> server into a virtualized environment, and I'm running into > performance > >> problems. VuFind is a Solr 1.4.1-based application with fairly > large > >> and complex records (many stored fields, many words per record). My > >> particular installation contains about a million records in the > index, > >> with a total index size around 6GB. > >> > > >> > The virtual environment has more RAM and better CPUs than the old > >> physical box, and I am satisfied that my Java environment is well- > >> tuned. My index is optimized. Searches that hit the cache respond > >> very well. The problem is that non-cached searches are very slow - > the > >> more keywords I add, the slower they get, to the point of taking 6- > 12 > >> seconds to come back with results on a quiet box and well over a > minute > >> under stress testing. (The old box still took a while for > equivalent > >> searches, but it was about twice as fast as the new one). > >> > > >> > My gut feeling is that disk access reading the index is the > >> bottleneck here, but I know little about the specifics of Solr's > >> internals, so it's entirely possible that my gut is wrong. Outside > >> testing does show that the the virtual environment's disk > performance > >> is not as good as the old physical server, especially when multiple > >> processes are trying to access the same file simultaneously. > >> > > >> > So, two basic questions: > >> > > >> > > >> > 1.) Would you agree that I'm dealing with a disk bottleneck, or > >> are there some other factors I should be considering? Any good > >> diagnostics I should be looking at? > >> > > >> > 2.) If the problem is disk access, is there anything I can tune > on > >> the Solr side to alleviate the problems? > >> > > >> > Thanks, > >> > Demian > >> > > >
