Another trick is to read in the parts of the index file that you search against: term dictionary and maybe a few others. (The Lucene wiki describes the various files.) That is, you copy the new index to the server and then say "cat files > /dev/null". This pre-caches the interesting files into memory.
This leads to: how large is you JVM and how much space do you leave to the OS? The OS is much better at managing memory against the hard disk that Solr/JVM is. The JVM should have enough memory to run your Solr comfortably without slowdowns, and that is the most it should get. You might find autowarming less useful than just picking a series of queries that warm what you want to get rolling early: sort on the fields you want, do a series of facet queries, search for words you get a lot, etc. Another problem is that you might be fighting garbage collection when switching from the old to new collection. Just shut down Solr, switch the index directory, cat the files mentioned above, and restart. On Fri, Jan 27, 2012 at 10:36 PM, Otis Gospodnetic <otis_gospodne...@yahoo.com> wrote: > Hi Dan, > > I think this may be your problem: > >> Every day we produce a new dataset of 40 GB and have to switch one for the >> othe > > If you really replace an index with a new index one a day, you throw away all > the hard work the OS has been doing to cache hot parts of your index in > memory. It takes it 30 minutes apparently in your cache to re-cache things. > Check the link in my signature. If you use that and if I'm right about > this, you will see a big spike in Disk Reads after you switch to the new > index. You want to minimize that spike. > > > So see if you can avoid replacing the whole index and if that is really not > doable, you can try warmup queries, but of course while you run them, if they > are expensive, they will hurt system performance somewhat. > > Otis > > ---- > Performance Monitoring SaaS for Solr - > http://sematext.com/spm/solr-performance-monitoring/index.html > > > > >>________________________________ >> From: dan sutton <danbsut...@gmail.com> >>To: solr-user <solr-user@lucene.apache.org> >>Sent: Friday, January 27, 2012 9:44 AM >>Subject: Solr Warm-up performance issues >> >>Hi List, >> >>We use Solr 4.0.2011.12.01.09.59.41 and have a dataset of roughly 40 GB. >>Every day we produce a new dataset of 40 GB and have to switch one for >>the other. >> >>Once the index switch over has taken place, it takes roughly 30 min for Solr >>to reach maximum performance. Are there any hardware or software solutions >>to reduce the warm-up time ? We tried warm-up queries but it didn't change >>much. >> >>Our hardware specs is: >> * Dell Poweredge 1950 >> * 2 x Quad-Core Xeon E5405 (2.00GHz) >> * 48 GB RAM >> * 2 x 146 GB SAS 3 Gb/s 15K RPM disk configured in RAID mirror >> >>One thing that does seem to take a long time is un-inverting a set of >>multivalued fields, are there any optimizations we might be able to >>use here? >> >>Thanks for your help. >>Dan >> >> >> -- Lance Norskog goks...@gmail.com
