Hi,
Thanks very much for your reply.
Because we need near real time search, we decide to use NRTCachingDirectory
instead of MMapDirectory.
Code to create Directory as follows :
Directory indexDir = FSDirectory.open(new File(indexDirName));
NRTCachingDirectory cachedFSDir = new NRTCachingDirectory(indexDir, 5.0,
60.0);
But I think that NRTCachingDirectory will only use RAMDirectory for caching
and use MMapDirectory to access index file on disk, right? The `top ` command
seems prove this, the VIRT memory of lucene server is 28.5G, and RES
memory is only 5G.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4004 root 20 0 28.5g 5.0g 49m S 2.0 65.6 140:34.50
java
Now our lucene server have indexed 2 million email and provide near real time
search service, and some times we can not commit the index because of
OutOfMemoryError, and we have to restart the JVM. By the way, we commit the
index for every 1000 email document.
Could you give me kindly give me some tips to solve this problem?
Thanks & Best Regards!
------------------ Original ------------------
From: "Uwe Schindler";<[email protected]>;
Date: Fri, Jun 27, 2014 08:36 PM
To: "java-user"<[email protected]>;
Subject: RE: About lucene memory consumption
Hi,
The number of byte[] instances and the total size shows that each byte[] is
approx. 1024 bytes long. This is exactly the size used by RAMDirectory for
allocated heap blocks.
So the important question: Do you use RAMDirectory to hold your index? This is
not recommended, it is better to use MMapDirectory. RAMDirectory is a class
made for testing lucene, not for production (does not scale well, is not
GC-friendly, and is therefore slow in most cases for large indexes). Also the
index is not persisted to disk. If you want an in-memory index, use a linux
tmpfs filesystem (ramdisk) and write your index to it (and use MMapDirectory
to access it).
To help you, give more information on how you use Lucene and its directory
implementations.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [email protected]
> -----Original Message-----
> From: 308181687 [mailto:[email protected]]
> Sent: Friday, June 27, 2014 10:42 AM
> To: java-user
> Subject: About lucene memory consumption
>
> Hi, all
>
>
> I fould that the memory consumption of my lucene server is abnormal, and
> “jmap -histo ${pid}” show that the class of byte[] consume almost all of the
> memory. Is there memory leak in my app? Why so many byte[] instances?
>
>
>
>
>
>
>
>
> The following is the top output of jmap:
>
>
> num #instances #bytes class name
> ----------------------------------------------
> 1: 1786575 1831556144 [B
> 2: 704618 80078064 [C
> 3: 839932 33597280 java.util.LinkedHashMap$Entry
> 4: 686770 21976640 java.lang.String
>
>
> ....
>
>
>
>
>
>
>
>
> Thanks & Best Regards!
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
.
