Hi,
I got a heap dump and use tool to analyze it, and then found that almost all
of the byte[] instances is indirectly referenced by SearcherManager.
Memory-path as follows:
byte[]
[Ljava/lang/Object
java/util/ArrayList
org/apache/lucene/store/RAMFile
org/apache/lucene/store/RAMInputStream
org/apache/lucene/store/Directory$SlicedIndexInput
org/apache/lucene/codecs/compressing/CompressingStoredFieldsReader
org/apache/lucene/index/SegmentCoreReaders
org/apache/lucene/index/SegmentReader
[Lorg/apache/lucene/index/SegmentReader;
org/apache/lucene/index/StandardDirectoryReader
org/apache/lucene/search/IndexSearcher
org/apache/lucene/search/SearcherManager
com/crm/lucene/LuceneMailIndex
By the way, there is only one SearcherManager instance. Why SearcherManager
consumes so much memory? Should i change my code to use MMapDirectory instead
of NRTCachingDirectory?
Thanks & Best Regards!
------------------ Original ------------------
From: "lubin";<308181...@qq.com>;
Date: Sat, Jun 28, 2014 02:03 PM
To: "java-user"<java-user@lucene.apache.org>;
Subject: Re:RE: About lucene memory consumption
Hi, Uwe
Actually I have alreadly used SearcherManager, and re-open it every 5
seconds. I notice that the number of byte[] instances is two times as the
the number of java.util.LinkedHashMap$Entry instances. It seems that there is
an big LinkedHashMap<byte[],byte[]> instance which cache something. Which
part of lucene might do this kind of caching behavior.
the top output of jmap as follows:
num #instances #bytes class name
----------------------------------------------
1: 1786575 1831556144 [B
2: 704618 80078064 [C
3: 839932 33597280 java.util.LinkedHashMap$Entry
4: 686770 21976640 java.lang.String
Thanks & Best Regards!
------------------ Original ------------------
From: "Uwe Schindler";<u...@thetaphi.de>;
Date: Fri, Jun 27, 2014 10:42 PM
To: "java-user"<java-user@lucene.apache.org>; ""<308181...@qq.com>;
Subject: Re:RE: About lucene memory consumption
Could it be that you forgot to close older IndexReaders after getting a new NRT
one? This would be a huge memory leak.
I recommend to use SearcherManager to handle real time reopen correctly.
Uwe
Am 27. Juni 2014 16:05:19 MESZ, schrieb 308181687 <308181...@qq.com>:
>Hi,
> Thanks very much for your reply.
>Because we need near real time search, we decide to use
>NRTCachingDirectory instead of MMapDirectory.
>
>
>
> Code to create Directory as follows :
>
>
> Directory indexDir = FSDirectory.open(new File(indexDirName));
>NRTCachingDirectory cachedFSDir = new NRTCachingDirectory(indexDir,
>5.0, 60.0);
>
>
>
>
>But I think that NRTCachingDirectory will only use RAMDirectory for
>caching and use MMapDirectory to access index file on disk, right? The
>`top ` command seems prove this, the VIRT memory of lucene server is
>28.5G, and RES memory is only 5G.
>
>
>PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>COMMAND
>4004 root 20 0 28.5g 5.0g 49m S 2.0 65.6
>140:34.50 java
>
>
>
>
>
>Now our lucene server have indexed 2 million email and provide near
>real time search service, and some times we can not commit the index
>because of OutOfMemoryError, and we have to restart the JVM. By the
>way, we commit the index for every 1000 email document.
>
>
> Could you give me kindly give me some tips to solve this problem?
>
>
>
>
>Thanks & Best Regards!
>
>
>
>
>
>
>
>
>
>
>------------------ Original ------------------
>From: "Uwe Schindler";<u...@thetaphi.de>;
>Date: Fri, Jun 27, 2014 08:36 PM
>To: "java-user"<java-user@lucene.apache.org>;
>
>Subject: RE: About lucene memory consumption
>
>
>
>Hi,
>
>The number of byte[] instances and the total size shows that each
>byte[] is approx. 1024 bytes long. This is exactly the size used by
>RAMDirectory for allocated heap blocks.
>So the important question: Do you use RAMDirectory to hold your index?
>This is not recommended, it is better to use MMapDirectory.
>RAMDirectory is a class made for testing lucene, not for production
>(does not scale well, is not GC-friendly, and is therefore slow in most
>cases for large indexes). Also the index is not persisted to disk. If
>you want an in-memory index, use a linux tmpfs filesystem (ramdisk) and
> write your index to it (and use MMapDirectory to access it).
>
>To help you, give more information on how you use Lucene and its
>directory implementations.
>
>Uwe
>
>-----
>Uwe Schindler
>H.-H.-Meier-Allee 63, D-28213 Bremen
>http://www.thetaphi.de
>eMail: u...@thetaphi.de
>
>
>> -----Original Message-----
>> From: 308181687 [mailto:308181...@qq.com]
>> Sent: Friday, June 27, 2014 10:42 AM
>> To: java-user
>> Subject: About lucene memory consumption
>>
>> Hi, all
>>
>>
>> I fould that the memory consumption of my lucene server is
>abnormal, and
>> “jmap -histo ${pid}” show that the class of byte[] consume almost all
>of the
>> memory. Is there memory leak in my app? Why so many byte[]
>instances?
>>
>>
>>
>>
>>
>>
>>
>>
>> The following is the top output of jmap:
>>
>>
>> num #instances #bytes class name
>> ----------------------------------------------
>> 1: 1786575 1831556144 [B
>> 2: 704618 80078064 [C
>> 3: 839932 33597280 java.util.LinkedHashMap$Entry
>> 4: 686770 21976640 java.lang.String
>>
>>
>> ....
>>
>>
>>
>>
>>
>>
>>
>>
>> Thanks & Best Regards!
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>.
--
Uwe Schindler
H.-H.-Meier-Allee 63, 28213 Bremen
http://www.thetaphi.de
