Hi, Thanks much for all suggestions. We're looking into reducing allocated heap size of Solr4 JVM.
We're using NRTCachingDirectoryFactory. Does it use MMapDirectory internally? Can someone please confirm? Would optimization help with performance? We did that in QA (took about 13 hours for 700 mil documents) Thanks! -----Original Message----- From: Roman Chyla [mailto:roman.ch...@gmail.com] Sent: Wednesday, February 12, 2014 3:17 PM To: solr-user@lucene.apache.org Subject: Re: Solr4 performance And perhaps one other, but very pertinent, recommendation is: allocate only as little heap as is necessary. By allocating more, you are working against the OS caching. To know how much is enough is bit tricky, though. Best, roman On Wed, Feb 12, 2014 at 2:56 PM, Shawn Heisey <s...@elyograg.org> wrote: > On 2/12/2014 12:07 PM, Greg Walters wrote: > >> Take a look at http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory- >> on-64bit.html as it's a pretty decent explanation of memory mapped >> files. I don't believe that the default configuration for solr is to use >> MMapDirectory but even if it does my understanding is that the entire file >> won't be forcibly cached by solr. The OS's filesystem cache should control >> what's actually in ram and the eviction process will depend on the OS. >> > > I only have a little bit to add. Here's the first thing that Uwe's blog > post (linked above) says: > > "Since version 3.1, *Apache Lucene*and *Solr *use MMapDirectoryby default > on 64bit Windows and Solaris systems; since version 3.3 also for 64bit > Linux systems." > > The default in Solr 4.x is NRTCachingDirectory, which uses MMapDirectory > by default under the hood. > > A summary about all this that should be relevant to the original question: > > It's the *operating system* that handles memory mapping, including any > caching that happens. Assuming that you don't have a badly configured > virtual machine setup, I'm fairly sure that only real memory gets used, > never swap space on the disk. If something else on the system makes a > memory allocation, the operating system will instantly give up memory used > for caching and mapping. One of the strengths of mmap is that it can't > exceed available resources unless it's used incorrectly. > > Thanks, > Shawn > >
