If you're seeing results, things must be OK. It's a little strange, though, I'm seeing warmup times of 1 on the trivial reload of the example documents.
But I wouldn't worry too much here. Those are pretty high autowarm counts, you might have room to reduce them but absent long autowarm times there's not much reason to mess with them... Best Erick On Mon, Jun 6, 2011 at 1:38 PM, Demian Katz <demian.k...@villanova.edu> wrote: > All of my cache autowarmCount settings are either 10000 or 50000. > maxWarmingSearchers is set to 2. I previously shared the contents of my > firstSearcher and newSearcher events -- just a "queries" array surrounded by > a standard-looking <listener> tag. The events are definitely firing -- in > addition to the measurable performance improvement they give me, I can > actually see them happening in the console output during startup. That seems > to cover every configuration option in my file that references warming in any > way, and it all looks reasonable to me. warmupTime remains consistently 0 in > the statistics display. Is there anything else I should be looking at? In > any case, I'm not too alarmed by this... it just seems a little strange. > > thanks, > Demian > >> -----Original Message----- >> From: Erick Erickson [mailto:erickerick...@gmail.com] >> Sent: Monday, June 06, 2011 11:59 AM >> To: solr-user@lucene.apache.org >> Subject: Re: Solr performance tuning - disk i/o? >> >> Polling interval was in reference to slaves in a multi-machine >> master/slave setup. so probably not >> a concern just at present. >> >> Warmup time of 0 is not particularly normal, I'm not quite sure what's >> going on there but you may >> want to look at firstsearcher, newsearcher and autowarm parameters in >> config.xml.. >> >> Best >> Erick >> >> On Mon, Jun 6, 2011 at 9:08 AM, Demian Katz <demian.k...@villanova.edu> >> wrote: >> > Thanks once again for the helpful suggestions! >> > >> > Regarding the selection of facet fields, I think publishDate (which >> is actually just a year) and callnumber-first (which is actually a very >> broad, high-level category) are okay. authorStr is an interesting >> problem: it's definitely a useful facet (when a user searches for an >> author, odds are good that they want the one who published the most >> books... i.e. a search for dickens will probably show Charles Dickens >> at the top of the facet list), but it has a long tail since there are >> many minor authors who have only published one or two books... Is >> there a possibility that the facet.mincount parameter could be helpful >> here, or does that have no impact on performance/memory footprint? >> > >> > Regarding polling interval for slaves, are you referring to a >> distributed Solr environment, or is this something to do with Solr's >> internals? We're currently a single-server environment, so I don't >> think I have to worry if it's related to a multi-server setup... but >> if it's something internal, could you point me to the right area of the >> admin panel to check my stats? I'm not seeing anything about polling >> on the statistics page. It's also a little strange that all of my >> warmupTime stats on searchers and caches are showing as 0 -- is that >> normal? >> > >> > thanks, >> > Demian >> > >> >> -----Original Message----- >> >> From: Erick Erickson [mailto:erickerick...@gmail.com] >> >> Sent: Friday, June 03, 2011 4:45 PM >> >> To: solr-user@lucene.apache.org >> >> Subject: Re: Solr performance tuning - disk i/o? >> >> >> >> Quick impressions: >> >> >> >> The faceting is usually best done on fields that don't have lots of >> >> unique >> >> values for three reasons: >> >> 1> It's questionable how much use to the user to have a gazillion >> >> facets. >> >> In the case of a unique field per document, in fact, it's >> useless. >> >> 2> resource requirements go up as a function of the number of unique >> >> terms. This is true for faceting and sorting. >> >> 3> warmup times grow the more terms have to be read into memory. >> >> >> >> >> >> Glancing at your warmup stuff, things like publishDate, authorStr >> and >> >> maybe >> >> callnumber-first are questionable. publishDate depends on how coarse >> >> the >> >> resolution is. If it's by day, that's not really much use. >> authorStr.. >> >> How many >> >> authors have more than one publication? Would this be better served >> by >> >> some >> >> kind of autosuggest rather than facets? callnumber-first... I don't >> >> really know, but >> >> if it's unique per document it's probably not something the user >> would >> >> find useful >> >> as a facet. >> >> >> >> The admin page will help you determine the number of unique terms >> per >> >> field, >> >> which may guide you whether or not to continue to facet on these >> >> fields. >> >> >> >> As Otis said, doing a sort on the fields during warmup will also >> help. >> >> >> >> Watch your polling interval for any slaves in relation to the warmup >> >> times. >> >> If your polling interval is shorter than the warmup times, you run a >> >> risk of >> >> "runaway warmups". >> >> >> >> As you've figured out, measuring responses to the first few queries >> >> doesn't >> >> always measure what you really need <G>.. >> >> >> >> I don't have the pages handy, but autowarming is a good topic to >> >> understand, >> >> so you might spend some time tracking it down. >> >> >> >> Best >> >> Erick >> >> >> >> On Fri, Jun 3, 2011 at 11:21 AM, Demian Katz >> >> <demian.k...@villanova.edu> wrote: >> >> > Thanks to you and Otis for the suggestions! Some more >> information: >> >> > >> >> > - Based on the Solr stats page, my caches seem to be working >> pretty >> >> well (few or no evictions, hit rates in the 75-80% range). >> >> > - VuFind is actually doing two Solr queries per search (one >> initial >> >> search followed by a supplemental spell check search -- I believe >> this >> >> is necessary because VuFind has two separate spelling indexes, one >> for >> >> shingled terms and one for single words). That is probably >> >> exaggerating the problem, though based on searches with debugQuery >> on, >> >> it looks like it's always the initial search (rather than the >> >> supplemental spelling search) that's consuming the bulk of the time. >> >> > - enableLazyFieldLoading is set to true. >> >> > - I'm retrieving 20 documents per page. >> >> > - My JVM settings: -server - >> >> Xloggc:/usr/local/vufind/solr/jetty/logs/gc.log -Xms4096m -Xmx4096m >> - >> >> XX:+UseParallelGC -XX:+UseParallelOldGC -XX:NewRatio=5 >> >> > >> >> > It appears that a large portion of my problem had to do with >> >> autowarming, a topic that I've never had a strong grasp on, though >> >> perhaps I'm finally learning (any recommended primer links would be >> >> welcome!). I did have some autowarming settings in solrconfig.xml >> (an >> >> arbitrary search for a bunch of random keywords in the newSearcher >> and >> >> firstSearcher events, plus autowarmCount settings on all of my >> caches). >> >> However, when I looked at the debugQuery output, I noticed that a >> huge >> >> amount of time was being wasted loading facets on the first search >> >> after restarting Solr, so I changed my newSearcher and firstSearcher >> >> events to this: >> >> > >> >> > <arr name="queries"> >> >> > <lst> >> >> > <str name="q">*:*</str> >> >> > <str name="start">0</str> >> >> > <str name="rows">10</str> >> >> > <str name="facet">true</str> >> >> > <str name="facet.mincount">1</str> >> >> > <str name="facet.field">collection</str> >> >> > <str name="facet.field">format</str> >> >> > <str name="facet.field">publishDate</str> >> >> > <str name="facet.field">callnumber-first</str> >> >> > <str name="facet.field">topic_facet</str> >> >> > <str name="facet.field">authorStr</str> >> >> > <str name="facet.field">language</str> >> >> > <str name="facet.field">genre_facet</str> >> >> > <str name="facet.field">era_facet</str> >> >> > <str name="facet.field">geographic_facet</str> >> >> > </lst> >> >> > </arr> >> >> > >> >> > Overall performance has now increased dramatically, and now the >> >> biggest bottleneck in the debug output seems to be the shingle spell >> >> checking! >> >> > >> >> > Any other suggestions are welcome, since I suspect there's still >> room >> >> to squeeze more performance out of the system, and I'm still not >> sure >> >> I'm making the most of autowarming... but this seems like a big >> step >> >> in the right direction. Thanks again for the help! >> >> > >> >> > - Demian >> >> > >> >> >> -----Original Message----- >> >> >> From: Erick Erickson [mailto:erickerick...@gmail.com] >> >> >> Sent: Friday, June 03, 2011 9:41 AM >> >> >> To: solr-user@lucene.apache.org >> >> >> Subject: Re: Solr performance tuning - disk i/o? >> >> >> >> >> >> This doesn't seem right. Here's a couple of things to try: >> >> >> 1> attach &debugQuery=on to your long-running queries. The QTime >> >> >> returned >> >> >> is the time taken to search, NOT including the time to load >> the >> >> >> docs. That'll >> >> >> help pinpoint whether the problem is the search itself, or >> >> >> assembling the >> >> >> documents. >> >> >> 2> Are you autowarming? If so, be sure it's actually done before >> >> >> querying. >> >> >> 3> Measure queries after the first few, particularly if you're >> >> sorting >> >> >> or >> >> >> faceting. >> >> >> 4> What are your JVM settings? How much memory do you have? >> >> >> 5> is <enableLazyFieldLoading> set to true in your >> solrconfig.xml? >> >> >> 6> How many docs are you returning? >> >> >> >> >> >> >> >> >> There's more, but that'll do for a start.... Let us know if you >> >> gather >> >> >> more data >> >> >> and it's still slow. >> >> >> >> >> >> Best >> >> >> Erick >> >> >> >> >> >> On Fri, Jun 3, 2011 at 8:44 AM, Demian Katz >> >> <demian.k...@villanova.edu> >> >> >> wrote: >> >> >> > Hello, >> >> >> > >> >> >> > I'm trying to move a VuFind installation from an ailing >> physical >> >> >> server into a virtualized environment, and I'm running into >> >> performance >> >> >> problems. VuFind is a Solr 1.4.1-based application with fairly >> >> large >> >> >> and complex records (many stored fields, many words per record). >> My >> >> >> particular installation contains about a million records in the >> >> index, >> >> >> with a total index size around 6GB. >> >> >> > >> >> >> > The virtual environment has more RAM and better CPUs than the >> old >> >> >> physical box, and I am satisfied that my Java environment is >> well- >> >> >> tuned. My index is optimized. Searches that hit the cache >> respond >> >> >> very well. The problem is that non-cached searches are very slow >> - >> >> the >> >> >> more keywords I add, the slower they get, to the point of taking >> 6- >> >> 12 >> >> >> seconds to come back with results on a quiet box and well over a >> >> minute >> >> >> under stress testing. (The old box still took a while for >> >> equivalent >> >> >> searches, but it was about twice as fast as the new one). >> >> >> > >> >> >> > My gut feeling is that disk access reading the index is the >> >> >> bottleneck here, but I know little about the specifics of Solr's >> >> >> internals, so it's entirely possible that my gut is wrong. >> Outside >> >> >> testing does show that the the virtual environment's disk >> >> performance >> >> >> is not as good as the old physical server, especially when >> multiple >> >> >> processes are trying to access the same file simultaneously. >> >> >> > >> >> >> > So, two basic questions: >> >> >> > >> >> >> > >> >> >> > 1.) Would you agree that I'm dealing with a disk bottleneck, >> or >> >> >> are there some other factors I should be considering? Any good >> >> >> diagnostics I should be looking at? >> >> >> > >> >> >> > 2.) If the problem is disk access, is there anything I can >> tune >> >> on >> >> >> the Solr side to alleviate the problems? >> >> >> > >> >> >> > Thanks, >> >> >> > Demian >> >> >> > >> >> > >> > >
