Re: Solr Warm-up performance issues
Another trick is to read in the parts of the index file that you search against: term dictionary and maybe a few others. (The Lucene wiki describes the various files.) That is, you copy the new index to the server and then say "cat files > /dev/null". This pre-caches the interesting files into memory. This leads to: how large is you JVM and how much space do you leave to the OS? The OS is much better at managing memory against the hard disk that Solr/JVM is. The JVM should have enough memory to run your Solr comfortably without slowdowns, and that is the most it should get. You might find autowarming less useful than just picking a series of queries that warm what you want to get rolling early: sort on the fields you want, do a series of facet queries, search for words you get a lot, etc. Another problem is that you might be fighting garbage collection when switching from the old to new collection. Just shut down Solr, switch the index directory, cat the files mentioned above, and restart. On Fri, Jan 27, 2012 at 10:36 PM, Otis Gospodnetic wrote: > Hi Dan, > > I think this may be your problem: > >> Every day we produce a new dataset of 40 GB and have to switch one for the >> othe > > If you really replace an index with a new index one a day, you throw away all > the hard work the OS has been doing to cache hot parts of your index in > memory. It takes it 30 minutes apparently in your cache to re-cache things. > Check the link in my signature. If you use that and if I'm right about > this, you will see a big spike in Disk Reads after you switch to the new > index. You want to minimize that spike. > > > So see if you can avoid replacing the whole index and if that is really not > doable, you can try warmup queries, but of course while you run them, if they > are expensive, they will hurt system performance somewhat. > > Otis > > > Performance Monitoring SaaS for Solr - > http://sematext.com/spm/solr-performance-monitoring/index.html > > > > >>________ >> From: dan sutton >>To: solr-user >>Sent: Friday, January 27, 2012 9:44 AM >>Subject: Solr Warm-up performance issues >> >>Hi List, >> >>We use Solr 4.0.2011.12.01.09.59.41 and have a dataset of roughly 40 GB. >>Every day we produce a new dataset of 40 GB and have to switch one for >>the other. >> >>Once the index switch over has taken place, it takes roughly 30 min for Solr >>to reach maximum performance. Are there any hardware or software solutions >>to reduce the warm-up time ? We tried warm-up queries but it didn't change >>much. >> >>Our hardware specs is: >> * Dell Poweredge 1950 >> * 2 x Quad-Core Xeon E5405 (2.00GHz) >> * 48 GB RAM >> * 2 x 146 GB SAS 3 Gb/s 15K RPM disk configured in RAID mirror >> >>One thing that does seem to take a long time is un-inverting a set of >>multivalued fields, are there any optimizations we might be able to >>use here? >> >>Thanks for your help. >>Dan >> >> >> -- Lance Norskog goks...@gmail.com
Re: Solr Warm-up performance issues
Hi Dan, I think this may be your problem: > Every day we produce a new dataset of 40 GB and have to switch one for the > othe If you really replace an index with a new index one a day, you throw away all the hard work the OS has been doing to cache hot parts of your index in memory. It takes it 30 minutes apparently in your cache to re-cache things. Check the link in my signature. If you use that and if I'm right about this, you will see a big spike in Disk Reads after you switch to the new index. You want to minimize that spike. So see if you can avoid replacing the whole index and if that is really not doable, you can try warmup queries, but of course while you run them, if they are expensive, they will hurt system performance somewhat. Otis Performance Monitoring SaaS for Solr - http://sematext.com/spm/solr-performance-monitoring/index.html > > From: dan sutton >To: solr-user >Sent: Friday, January 27, 2012 9:44 AM >Subject: Solr Warm-up performance issues > >Hi List, > >We use Solr 4.0.2011.12.01.09.59.41 and have a dataset of roughly 40 GB. >Every day we produce a new dataset of 40 GB and have to switch one for >the other. > >Once the index switch over has taken place, it takes roughly 30 min for Solr >to reach maximum performance. Are there any hardware or software solutions >to reduce the warm-up time ? We tried warm-up queries but it didn't change >much. > >Our hardware specs is: > * Dell Poweredge 1950 > * 2 x Quad-Core Xeon E5405 (2.00GHz) > * 48 GB RAM > * 2 x 146 GB SAS 3 Gb/s 15K RPM disk configured in RAID mirror > >One thing that does seem to take a long time is un-inverting a set of >multivalued fields, are there any optimizations we might be able to >use here? > >Thanks for your help. >Dan > > >
Re: Solr Warm-up performance issues
You say warming queries didn't help? How do those look like? Make sure you facet and sort in all of the fields that your application allow faceting/sorting. The same with the filters. Uninversion of fields is done only when you commit, but warming queries should help you here. Tomás On Fri, Jan 27, 2012 at 11:44 AM, dan sutton wrote: > Hi List, > > We use Solr 4.0.2011.12.01.09.59.41 and have a dataset of roughly 40 GB. > Every day we produce a new dataset of 40 GB and have to switch one for > the other. > > Once the index switch over has taken place, it takes roughly 30 min for > Solr > to reach maximum performance. Are there any hardware or software solutions > to reduce the warm-up time ? We tried warm-up queries but it didn't change > much. > > Our hardware specs is: > * Dell Poweredge 1950 > * 2 x Quad-Core Xeon E5405 (2.00GHz) > * 48 GB RAM > * 2 x 146 GB SAS 3 Gb/s 15K RPM disk configured in RAID mirror > > One thing that does seem to take a long time is un-inverting a set of > multivalued fields, are there any optimizations we might be able to > use here? > > Thanks for your help. > Dan >
RE: Solr Warm-up performance issues
Dan, I can suggest a solution that should help. VeloBit enables you to add SSDs to your servers as a cache (SSD will cost you $200, per server should be enough). Then, assuming a 100MB/s read speed from your SAS disks, you can read 50GB data into the VeloBit HyperCache cache in about 9 mins (this happens automatically, all you need to do is add the SSD to your server and install Velobit one time, which takes 2 minutes). Solr should run much faster after that. The added benefit of the solution is that you would have also boosted the steady state performance by 4x. Let me know if you are interested in trying it out and I'll set you up to talk with my engineers. Best regards, Peter Velikin VP Online Marketing, VeloBit, Inc. pe...@velobit.com tel. 978-263-4800 mob. 617-306-7165 VeloBit provides plug & play SSD caching software that dramatically accelerates applications at a remarkably low cost. The software installs seamlessly in less than 10 minutes and automatically tunes for fastest application speed. Visit www.velobit.com for details. -Original Message- From: dan sutton [mailto:danbsut...@gmail.com] Sent: Friday, January 27, 2012 9:44 AM To: solr-user Subject: Solr Warm-up performance issues Hi List, We use Solr 4.0.2011.12.01.09.59.41 and have a dataset of roughly 40 GB. Every day we produce a new dataset of 40 GB and have to switch one for the other. Once the index switch over has taken place, it takes roughly 30 min for Solr to reach maximum performance. Are there any hardware or software solutions to reduce the warm-up time ? We tried warm-up queries but it didn't change much. Our hardware specs is: * Dell Poweredge 1950 * 2 x Quad-Core Xeon E5405 (2.00GHz) * 48 GB RAM * 2 x 146 GB SAS 3 Gb/s 15K RPM disk configured in RAID mirror One thing that does seem to take a long time is un-inverting a set of multivalued fields, are there any optimizations we might be able to use here? Thanks for your help. Dan
Solr Warm-up performance issues
Hi List, We use Solr 4.0.2011.12.01.09.59.41 and have a dataset of roughly 40 GB. Every day we produce a new dataset of 40 GB and have to switch one for the other. Once the index switch over has taken place, it takes roughly 30 min for Solr to reach maximum performance. Are there any hardware or software solutions to reduce the warm-up time ? We tried warm-up queries but it didn't change much. Our hardware specs is: * Dell Poweredge 1950 * 2 x Quad-Core Xeon E5405 (2.00GHz) * 48 GB RAM * 2 x 146 GB SAS 3 Gb/s 15K RPM disk configured in RAID mirror One thing that does seem to take a long time is un-inverting a set of multivalued fields, are there any optimizations we might be able to use here? Thanks for your help. Dan