Hu Uwe, Thanks Wwe, Have you checked the Bug in JRE for mmapDirectory?. I was mentioning this, This is posted in Oracle site, and the API doc. They accept this as a bug, have you seen this?.
“MMapDirectory<http://lucene.apache.org/java/3_0_2/api/core/org/apache/lucene/store/MMapDirectory.html>uses memory-mapped IO when reading. This is a good choice if you have plenty of virtual memory relative to your index size, eg if you are running on a 64 bit JRE, or you are running on a 32 bit JRE but your index sizes are small enough to fit into the virtual memory space. Java has currently the limitation of not being able to unmap files from user code. The files are unmapped, when GC releases the byte buffers. *Due to this bug<http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038>in Sun's JRE, MMapDirectory's **IndexInput.close()*<http://lucene.apache.org/java/3_0_2/api/core/org/apache/lucene/store/IndexInput.html#close%28%29> * is unable to close the underlying OS file handle. Only when GC finally collects the underlying objects, which could be quite some time later, will the file handle be closed*. *This will consume additional transient disk usage*: on Windows, attempts to delete or overwrite the files will result in an exception; on other platforms, which typically have a "delete on last close" semantics, while such operations will succeed, the bytes are still consuming space on disk. For many applications this limitation is not a problem (e.g. if you have plenty of disk space, and you don't rely on overwriting files on Windows) but it's still an important limitation to be aware of. This class supplies a (possibly dangerous) workaround mentioned in the bug report, which may fail on non-Sun JVMs. “ Thanks, On Mon, Jul 23, 2012 at 4:13 AM, Uwe Schindler <u...@thetaphi.de> wrote: > It is hopeless to talk to both of you, you don't understand virtual memory: > > > I get a similar situation using Windows 2008 and Solr 3.6. Memory using > > mmap=is never released. Even if I turn off traffic and commit and do a > manual > > gc= If the size of the index is 3gb then memory used will be heap + 3gb > of > > sha=ed used. If I use a 6gb index I get heap + 6gb. > > That is expected, but we are talking not about allocated physical memory, > we > are talking about allocated ADDRESS SPACE and you have 2^47 of that on > 64bit > platforms. There is no physical memory wasted or allocated - please read > the > blog post a third, forth, fifth... or tenth time, until it is obvious. You > should also go back to school and take a course on system programming and > operating system kernels. Every CS student gets that taught in his first > year (at least in Germany). > > Java's GC has nothing to do with that - as long as the index is open, > ADDRESS SPACE is assigned. We are talking not about memory nor Java heap > space. > > > If I turn off > > MMapDirectory=actory it goes back down. When is the MMap supposed to > > release memory ? It o=ly does it on JVM restart now. > > Can you please stop spreading nonsense about MMapDirectory with no > knowledge > behind? http://www.linuxatemyram.com/ - Also applies to Windows. > > Uwe > > > Bill Bell > > Sent from mobile > > > > > > On Jul 22, 2012, at 6:21 AM, geetha anjali <anjaliprabh...@gmail.com> > > wrote:= > > > It happens in 3.6, for this reasons I thought of moving to solandra. > > > If I do a commit, the all documents are persisted with out any issues. > > > There is no issues in terms of any functionality, but only this > > > happens i= increase in physical RAM, goes higher and higher and stop > > > at maximum and i= never comes down. > > > > > > Thanks > > > > > > On Sun, Jul 22, 2012 at 3:38 AM, Lance Norskog <goks...@gmail.com> > > wrote: > > > > > >> Interesting. Which version of Solr is this? What happens if you do a > > >> commit? > > >> > > >> On Sat, Jul 21, 2012 at 8:01 AM, geetha anjali > > <anjaliprabh...@gmail.com>=>> wrote: > > >>> Hi uwe, > > >>> Great to know. We have files indexing 10000/min. After 30 mins I see > > >>> all=>>> my physical memory say its 100 percentage used(windows). On > > >>> deep investigation found that mmap is not releasing os files handles. > Do > > you find this behaviour? > > >>> > > >>> Thanks > > >>> > > >>> On 20 Jul 2012 14:04, "Uwe Schindler" <u...@thetaphi.de> wrote: > > >>> > > >>> Hi Bill, > > >>> > > >>> MMapDirectory uses the file system cache of your operating system, > > >>> which=>> has following consequences: In Linux, top & free should > > >>> normally report only=>>> *few* free memory, because the O/S uses all > > >>> memory not allocated by applications to cache disk I/O (and shows it > > >>> as allocated, so having 0% > > >> free > > >>> memory is just normal on Linux and also Windows). If you have other > > >>> applications or Lucene/Solr itself that allocate lot's of heap space > > >>> or > > >>> malloc() a lot, then you are reducing free physical memory, so > > >>> reducing > > >> fs > > >>> cache. This depends also on your swappiness parameter (if swappiness > > >>> is higher, inactive processes are swapped out easier, default is 60% > > >>> on > > >> linux - > > >>> freeing more space for FS cache - the backside is of course that > > >>> maybe in-memory structures of Lucene and other applications get pages > > out). > > >>> > > >>> You will only see no paging at all if all memory allocated all > > >> applications > > >>> + all mmapped files fit into memory. But paging in/out the mmapped > > >>> + Lucen= > > >>> index is muuuuuch cheaper than using SimpleFSDirectory or > > >> NIOFSDirectory. If > > >>> you use SimpleFS or NIO and your index is not in FS cache, it will > > >>> also > > >> read > > >>> it from physical disk again, so where is the difference. Paging is > > >> actually > > >>> cheaper as no syscalls are involved. > > >>> > > >>> If you want as much as possible of your index in physical RAM, copy > > >>> it t= /dev/null regularily and buy more RUM :-) > > >>> > > >>> > > >>> ----- > > >>> Uwe Schindler > > >>> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > > >>> eMail: uwe@thetaphi... > > >>> > > >>>> From: Bill Bell [mailto:billnb...@gmail.com] > > >>>> Sent: Friday, July 20, 2012 5:17 AM > > >>>> Subject: Re: ... > > >>>> s=op using it? The least used memory will be removed from the OS > > >>>> automaticall=? Isee some paging. Wouldn't paging slow down the > > queryi=g? > > >>> > > >>>> > > >>>> My index is 10gb and every 8 hours we get most of it in shared > memory. > > >> The > > >>>> m=mory is 99 percent used, and that does not leave any room for > > >>>> other=>>> apps. = > > >>> > > >>>> Other implications? > > >>>> > > >>>> Sent from my mobile device > > >>>> 720-256-8076 > > >>>> > > >>>> On Jul 19, 2012, at 9:49 A... > > >>>> H=ap space or free system RAM: > > >>> > > >>>>> > > >>>>> > > >> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.ht > > >> m > > >>>>> l > > >>>>> > > >>>>> Uwe > > >>>>> ... > > >>>>>> use i= since you might run out of memory on large indexes right? > > >>> > > >>>>>> > > >>>>>> Here is how I got iSimpleFSDirectoryFactory to work. Just set - > > >>>>>> Dsolr.directoryFactor... > > >>>>>> set it=all up with a helper in solrconfig.xml... > > >>> > > >>>>>> > > >>>>>> if (Constants.WINDOWS) { > > >>>>>> if (MMapDirectory.UNMAP_SUPPORTED && Constants.JRE_IS_64... > > >> > > >> > > >> > > >> -- > > >> Lance Norskog > > >> goks...@gmail.com > > >> > > >
