Hi list,
while monitoring my solr 3.6.1 installation I recognized an increase of memory
usage
in OldGen JVM heap on my slave. I decided to force Full GC from jvisualvm and
send optimize to the already optimized slave index. Normally this helps because
I have monitored this issue over the past. But not this time. The Full GC
didn't free any memory. So I decided to take a heap dump and see what
MemoryAnalyzer
is showing. The heap dump is about 23 GB in size.
1.)
Report Top consumers - Biggest Objects:
Total: 12.3 GB
org.apache.lucene.search.FieldCacheImpl : 8.1 GB
class java.lang.ref.Finalizer : 2.1 GB
org.apache.solr.util.ConcurrentLRUCache : 1.5 GB
org.apache.lucene.index.ReadOnlySegmentReader : 622.5 MB
...
As you can see, Finalizer has already reached 2.1 GB!!!
* java.util.concurrent.ConcurrentHashMap$Segment[16] @ 0x37b056fd0
* segments java.util.concurrent.ConcurrentHashMap @ 0x39b02d268
* map org.apache.solr.util.ConcurrentLRUCache @ 0x398f33c30
* referent java.lang.ref.Finalizer @ 0x37affa810
* next java.lang.ref.Finalizer @ 0x37affa838
...
Seams to be org.apache.solr.util.ConcurrentLRUCache
The attributes are:
Type | Name | Value
-----------------------------
boolean| isDestroyed | true
-----------------------------
ref | cleanupThread | null
----------------------------
ref | evictionListener | null
-------------------------------
long | oldestEntry | 0
----------------------------------
int | acceptableWaterMark | 9500
--------------------------------------------------------------------------
ref | stats | org.apache.solr.util.ConcurrentLRUCache$Stats @
0x37b074dc8
------------------------
boolean| islive | true
-------------------------------------
boolean| newThreadForCleanup | false
----------------------------
boolean| isCleaning | false
--------------------------------------------------------------------------------
ref | markAndSweepLock | java.util.concurrent.locks.ReentrantLock @
0x39bf63978
-----------------------------
int | lowerWaterMark | 9000
-----------------------------
int | upperWaterMark | 10000
-----------------------------
ref | map | java.util.concurrent.ConcurrentHashMap @ 0x39b02d268
------------------------------------------------------------------
2.)
While searching for open files and their references I noticed that there are
references to
index files which are already deleted from disk.
E.g. recent index files are "data/index/_2iqw.frq" and "data/index/_2iqx.frq".
But I also see references to "data/index/_2hid.frq" which are quite old and are
deleted way back
from earlier replications.
I have to analyze this a bit deeper.
So far my report, I go on analyzing this huge heap dump.
If you need any other info or even the heap dump, let me know.
Regards
Bernd
