Be careful what you think is being used by Solr since Lucene uses MMapDirectories under the covers, and this means you might be seeing virtual memory. See Uwe's excellent blog here: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Best, Erick On Tue, Feb 24, 2015 at 5:02 PM, Walter Underwood <wun...@wunderwood.org> wrote: > The other memory is used by the OS as file buffers. All the important parts > of the on-disk search index are buffered in memory. When the Solr process > wants a block, it is already right there, no delays for disk access. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > On Feb 24, 2015, at 4:45 PM, Tang, Rebecca <rebecca.t...@ucsf.edu> wrote: > >> We gave the machine 180G mem to see if it improves performance. However, >> after we increased the memory, Solr started using only 5% of the physical >> memory. It has always used 90-something%. >> >> What could be causing solr to not grab all the physical memory (grabbing >> so little of the physical memory)? >> >> Rebecca Tang >> Applications Developer, UCSF CKM >> Industry Documents Digital Libraries >> E: rebecca.t...@ucsf.edu >> >> On 2/24/15 12:44 PM, "Shawn Heisey" <apa...@elyograg.org> wrote: >> >>> On 2/24/2015 1:09 PM, Tang, Rebecca wrote: >>>> Our solr index used to perform OK on our beta production box (anywhere >>>> between 0-3 seconds to complete any query), but today I noticed that the >>>> performance is very bad (queries take between 12 15 seconds). >>>> >>>> I haven't updated the solr index configuration >>>> (schema.xml/solrconfig.xml) lately. All that's changed is the data ‹ >>>> every month, I rebuild the solr index from scratch and deploy it to the >>>> box. We will eventually go to incremental builds. But for now, all >>>> indexes are built from scratch. >>>> >>>> Here are the stats: >>>> Solr index size 183G >>>> Documents in index 14364201 >>>> We just have single solr box >>>> It has 100G memory >>>> 500G Harddrive >>>> 16 cpus >>> >>> The bottom line on this problem, and I'm sure it's not something you're >>> going to want to hear: You don't have enough memory available to cache >>> your index. I'd plan on at least 192GB of RAM for an index this size, >>> and 256GB would be better. >>> >>> Depending on the exact index schema, the nature of your queries, and how >>> large your Java heap for Solr is, 100GB of RAM could be enough for good >>> performance on an index that size ... or it might be nowhere near >>> enough. I would imagine that one of two things is true here, possibly >>> both: 1) Your queries are very complex and involve accessing a very >>> large percentage of the index data. 2) Your Java heap is enormous, >>> leaving very little RAM for the OS to automatically cache the index. >>> >>> Adding more memory to the machine, if that's possible, might fix some of >>> the problems. You can find a discussion of the problem here: >>> >>> http://wiki.apache.org/solr/SolrPerformanceProblems >>> >>> If you have any questions after reading that wiki article, feel free to >>> ask them. >>> >>> Thanks, >>> Shawn >>> >> >
