Could it be that you forgot to close older IndexReaders after getting a new NRT one? This would be a huge memory leak.
I recommend to use SearcherManager to handle real time reopen correctly. Uwe Am 27. Juni 2014 16:05:19 MESZ, schrieb 308181687 <308181...@qq.com>: >Hi, > Thanks very much for your reply. >Because we need near real time search, we decide to use >NRTCachingDirectory instead of MMapDirectory. > > > > Code to create Directory as follows : > > > Directory indexDir = FSDirectory.open(new File(indexDirName)); >NRTCachingDirectory cachedFSDir = new NRTCachingDirectory(indexDir, >5.0, 60.0); > > > > >But I think that NRTCachingDirectory will only use RAMDirectory for >caching and use MMapDirectory to access index file on disk, right? The >`top ` command seems prove this, the VIRT memory of lucene server is >28.5G, and RES memory is only 5G. > > >PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >COMMAND >4004 root 20 0 28.5g 5.0g 49m S 2.0 65.6 >140:34.50 java > > > > > >Now our lucene server have indexed 2 million email and provide near >real time search service, and some times we can not commit the index >because of OutOfMemoryError, and we have to restart the JVM. By the >way, we commit the index for every 1000 email document. > > > Could you give me kindly give me some tips to solve this problem? > > > > >Thanks & Best Regards! > > > > > > > > > > >------------------ Original ------------------ >From: "Uwe Schindler";<u...@thetaphi.de>; >Date: Fri, Jun 27, 2014 08:36 PM >To: "java-user"<java-user@lucene.apache.org>; > >Subject: RE: About lucene memory consumption > > > >Hi, > >The number of byte[] instances and the total size shows that each >byte[] is approx. 1024 bytes long. This is exactly the size used by >RAMDirectory for allocated heap blocks. >So the important question: Do you use RAMDirectory to hold your index? >This is not recommended, it is better to use MMapDirectory. >RAMDirectory is a class made for testing lucene, not for production >(does not scale well, is not GC-friendly, and is therefore slow in most >cases for large indexes). Also the index is not persisted to disk. If >you want an in-memory index, use a linux tmpfs filesystem (ramdisk) and > write your index to it (and use MMapDirectory to access it). > >To help you, give more information on how you use Lucene and its >directory implementations. > >Uwe > >----- >Uwe Schindler >H.-H.-Meier-Allee 63, D-28213 Bremen >http://www.thetaphi.de >eMail: u...@thetaphi.de > > >> -----Original Message----- >> From: 308181687 [mailto:308181...@qq.com] >> Sent: Friday, June 27, 2014 10:42 AM >> To: java-user >> Subject: About lucene memory consumption >> >> Hi, all >> >> >> I fould that the memory consumption of my lucene server is >abnormal, and >> “jmap -histo ${pid}” show that the class of byte[] consume almost all >of the >> memory. Is there memory leak in my app? Why so many byte[] >instances? >> >> >> >> >> >> >> >> >> The following is the top output of jmap: >> >> >> num #instances #bytes class name >> ---------------------------------------------- >> 1: 1786575 1831556144 [B >> 2: 704618 80078064 [C >> 3: 839932 33597280 java.util.LinkedHashMap$Entry >> 4: 686770 21976640 java.lang.String >> >> >> .... >> >> >> >> >> >> >> >> >> Thanks & Best Regards! > > >--------------------------------------------------------------------- >To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >For additional commands, e-mail: java-user-h...@lucene.apache.org > >. -- Uwe Schindler H.-H.-Meier-Allee 63, 28213 Bremen http://www.thetaphi.de
