I get a similar situation using Windows 2008 and Solr 3.6. Memory using mmap is never released. Even if I turn off traffic and commit and do a manual gc. If the size of the index is 3gb then memory used will be heap + 3gb of shared used. If I use a 6gb index I get heap + 6gb. If I turn off MMapDirectoryFactory it goes back down. When is the MMap supposed to release memory ? It only does it on JVM restart now.
Bill Bell Sent from mobile On Jul 22, 2012, at 6:21 AM, geetha anjali <anjaliprabh...@gmail.com> wrote: > It happens in 3.6, for this reasons I thought of moving to solandra. > If I do a commit, the all documents are persisted with out any issues. > There is no issues in terms of any functionality, but only this happens is > increase in physical RAM, goes higher and higher and stop at maximum and it > never comes down. > > Thanks > > On Sun, Jul 22, 2012 at 3:38 AM, Lance Norskog <goks...@gmail.com> wrote: > >> Interesting. Which version of Solr is this? What happens if you do a >> commit? >> >> On Sat, Jul 21, 2012 at 8:01 AM, geetha anjali <anjaliprabh...@gmail.com> >> wrote: >>> Hi uwe, >>> Great to know. We have files indexing 10000/min. After 30 mins I see all >>> my physical memory say its 100 percentage used(windows). On deep >>> investigation found that mmap is not releasing os files handles. Do you >>> find this behaviour? >>> >>> Thanks >>> >>> On 20 Jul 2012 14:04, "Uwe Schindler" <u...@thetaphi.de> wrote: >>> >>> Hi Bill, >>> >>> MMapDirectory uses the file system cache of your operating system, which >> has >>> following consequences: In Linux, top & free should normally report only >>> *few* free memory, because the O/S uses all memory not allocated by >>> applications to cache disk I/O (and shows it as allocated, so having 0% >> free >>> memory is just normal on Linux and also Windows). If you have other >>> applications or Lucene/Solr itself that allocate lot's of heap space or >>> malloc() a lot, then you are reducing free physical memory, so reducing >> fs >>> cache. This depends also on your swappiness parameter (if swappiness is >>> higher, inactive processes are swapped out easier, default is 60% on >> linux - >>> freeing more space for FS cache - the backside is of course that maybe >>> in-memory structures of Lucene and other applications get pages out). >>> >>> You will only see no paging at all if all memory allocated all >> applications >>> + all mmapped files fit into memory. But paging in/out the mmapped Lucene >>> index is muuuuuch cheaper than using SimpleFSDirectory or >> NIOFSDirectory. If >>> you use SimpleFS or NIO and your index is not in FS cache, it will also >> read >>> it from physical disk again, so where is the difference. Paging is >> actually >>> cheaper as no syscalls are involved. >>> >>> If you want as much as possible of your index in physical RAM, copy it to >>> /dev/null regularily and buy more RUM :-) >>> >>> >>> ----- >>> Uwe Schindler >>> H.-H.-Meier-Allee 63, D-28213 Bremen >>> http://www.thetaphi.de >>> eMail: uwe@thetaphi... >>> >>>> From: Bill Bell [mailto:billnb...@gmail.com] >>>> Sent: Friday, July 20, 2012 5:17 AM >>>> Subject: Re: ... >>>> s=op using it? The least used memory will be removed from the OS >>>> automaticall=? Isee some paging. Wouldn't paging slow down the querying? >>> >>>> >>>> My index is 10gb and every 8 hours we get most of it in shared memory. >> The >>>> m=mory is 99 percent used, and that does not leave any room for other >>> apps. = >>> >>>> Other implications? >>>> >>>> Sent from my mobile device >>>> 720-256-8076 >>>> >>>> On Jul 19, 2012, at 9:49 A... >>>> H=ap space or free system RAM: >>> >>>>> >>>>> >> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.htm >>>>> l >>>>> >>>>> Uwe >>>>> ... >>>>>> use i= since you might run out of memory on large indexes right? >>> >>>>>> >>>>>> Here is how I got iSimpleFSDirectoryFactory to work. Just set - >>>>>> Dsolr.directoryFactor... >>>>>> set it=all up with a helper in solrconfig.xml... >>> >>>>>> >>>>>> if (Constants.WINDOWS) { >>>>>> if (MMapDirectory.UNMAP_SUPPORTED && Constants.JRE_IS_64... >> >> >> >> -- >> Lance Norskog >> goks...@gmail.com >>
