[
https://issues.apache.org/jira/browse/CASSANDRA-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161451#comment-13161451
]
MaHaiyang edited comment on CASSANDRA-3537 at 12/2/11 6:52 AM:
---------------------------------------------------------------
Heap memory can not be GC again in my lasted test , although I have set jamm
correctly . Jmap the memory ,find the same thing ,too many Message objects and
ralated objects in the memory !
I have empty all memtables ,in normal condition , heap memory used not more
than 0.5G ,but now more than 2.8G . I think only memory leak will result this .
{code:xml}
Object Histogram:
num #instances #bytes Class description
--------------------------------------------------------------------------
1: 9101138 927302160 byte[]
2: 9998059 479906832 java.nio.HeapByteBuffer
3: 875398 221844128 long[]
4: 4302698 137686336 org.apache.cassandra.db.Column
5: 869859 137476712 java.lang.Object[]
6: 1794458 101193096 int[]
7: 1515578 60623120 java.math.BigInteger
8: 834628 46739168
org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
9: 1644330 39463920
java.util.concurrent.ConcurrentSkipListMap$Node
10: 1461787 35082888 org.apache.cassandra.db.DecoratedKey
11: 513576 24651648 java.util.Hashtable
12: 741037 24559960 char[]
13: 1512396 24198336 org.apache.cassandra.dht.BigIntegerToken
14: 741097 23715104 java.lang.String
15: 732184 17572416
org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
16: 513594 15637344 java.util.Hashtable$Entry[]
17: 278209 15579704
org.cliffc.high_scale_lib.NonBlockingHashMap$CHM
18: 462641 14804512 java.net.Inet4Address
19: 578801 13891224
java.util.concurrent.ConcurrentSkipListMap$Index
20: 834627 13354032 org.cliffc.high_scale_lib.Counter
21: 556415 13353960 org.apache.cassandra.db.ReadResponse
22: 267483 12839184
org.apache.cassandra.service.ReadCallback
23: 513250 12318000 org.apache.cassandra.net.Message
24: 513248 12317952 org.apache.cassandra.net.Header
25: 501596 12038304
org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
26: 244644 11742912
org.apache.cassandra.db.ThreadSafeSortedColumns
27: 365024 11680768
java.util.concurrent.ConcurrentSkipListMap$HeadIndex
28: 464800 11155200 java.util.Collections$SynchronizedSet
29: 278209 11128360
org.cliffc.high_scale_lib.NonBlockingHashMap
30: 342741 10967712
org.apache.cassandra.utils.ExpiringMap$CacheableObject
31: 199478 9574944 org.apache.cassandra.db.SliceFromReadCommand
32: 278218 8902976
java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
33: 348179 8356296 org.apache.cassandra.db.ColumnFamily
34: 342741 8225784 org.apache.cassandra.net.CallbackInfo
35: 255241 8167712 org.apache.cassandra.db.ArrayBackedSortedColumns
36: 334724 8033376 java.util.ArrayList
37: 499963 7999408 java.util.concurrent.atomic.AtomicReference
38: 319779 7674696 org.apache.cassandra.db.Row
39: 187484 7499360 org.apache.cassandra.db.Memtable$6
40: 464794 7436704 java.util.Hashtable$KeySet
41: 187496 5999872 java.util.TreeMap$KeyIterator
42: 372748 5963968 java.util.concurrent.atomic.AtomicInteger
43: 236877 5685048 org.apache.cassandra.db.filter.QueryPath
44: 236733 5681592 org.apache.cassandra.service.RowDigestResolver
45: 300385 4806160 org.apache.cassandra.utils.SimpleCondition
46: 30726 4245920 * ConstMethodKlass
47: 30726 3696240 * MethodKlass
48: 151726 3641424 org.apache.cassandra.db.SuperColumn
49: 3134 3350536 * ConstantPoolKlass
50: 95622 3059904
java.util.concurrent.locks.ReentrantLock$NonfairSync
51: 45910 2497952 * SymbolKlass
52: 3134 2276040 * InstanceKlassKlass
53: 94544 2269056 java.util.concurrent.LinkedBlockingQueue$Node
54: 93130 2235120
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
55: 133398 2134368
java.util.concurrent.ConcurrentSkipListMap$Values
56: 2813 2099368 * ConstantPoolCacheKlass
57: 37698 1809504 java.util.TreeMap
58: 37783 1511320 java.util.TreeMap$Entry
59: 93602 1497632 java.util.concurrent.locks.ReentrantLock
60: 30768 1476864 java.util.concurrent.LinkedBlockingQueue
61: 85156 1362496
java.util.concurrent.ConcurrentSkipListMap$EntrySet
62: 41451 1326432 org.apache.cassandra.service.RowRepairResolver
63: 41141 1316512 java.util.concurrent.ConcurrentHashMap$HashEntry
64: 32885 1315400
org.apache.cassandra.service.WriteResponseHandler
65: 3110 1277672 * MethodDataKlass
66: 39780 1272960
com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
67: 31556 1262240 org.apache.cassandra.net.AsyncResult
68: 30745 1229800 org.apache.cassandra.db.RangeSliceCommand
69: 37628 1204096 org.apache.cassandra.db.SliceByNamesReadCommand
70: 44528 1068672
com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$WeightedValue
71: 31974 1023168
java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
72: 41435 994440 org.apache.cassandra.service.AsyncRepairCallback
73: 30821 986272 org.apache.cassandra.thrift.SliceRange
74: 30756 984192 java.util.RandomAccessSubList
75: 30745 983840
org.apache.cassandra.service.RangeSliceResponseResolver
76: 29115 931680 org.apache.cassandra.db.DeletedColumn
77: 21676 867040 org.apache.cassandra.db.ExpiringColumn
78: 35339 848136 java.lang.Long
79: 35204 844896 org.apache.cassandra.utils.Pair
80: 30924 742176 java.util.BitSet
81: 30821 739704 org.apache.cassandra.thrift.SlicePredicate
82: 30787 738888 org.apache.cassandra.dht.Bounds
83: 37673 602768 java.util.TreeSet
84: 37654 602464 java.util.TreeMap$KeySet
85: 31598 505568 java.util.concurrent.atomic.AtomicBoolean
86: 6575 420800 java.nio.DirectByteBufferR
87: 2004 416400
java.util.concurrent.ConcurrentHashMap$HashEntry[]
88: 3433 357032 java.lang.Class
89: 8976 287232 java.lang.ref.WeakReference
90: 4999 269008 short[]{code}
was (Author: jonma):
Heap memory can not be GC again in my lasted test , although I have set
jamm correctly . Jmap the memory ,find the same thing ,too many Message
objects and ralated objects in the memory !
{code:xml}
Object Histogram:
num #instances #bytes Class description
--------------------------------------------------------------------------
1: 9101138 927302160 byte[]
2: 9998059 479906832 java.nio.HeapByteBuffer
3: 875398 221844128 long[]
4: 4302698 137686336 org.apache.cassandra.db.Column
5: 869859 137476712 java.lang.Object[]
6: 1794458 101193096 int[]
7: 1515578 60623120 java.math.BigInteger
8: 834628 46739168
org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
9: 1644330 39463920
java.util.concurrent.ConcurrentSkipListMap$Node
10: 1461787 35082888 org.apache.cassandra.db.DecoratedKey
11: 513576 24651648 java.util.Hashtable
12: 741037 24559960 char[]
13: 1512396 24198336 org.apache.cassandra.dht.BigIntegerToken
14: 741097 23715104 java.lang.String
15: 732184 17572416
org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
16: 513594 15637344 java.util.Hashtable$Entry[]
17: 278209 15579704
org.cliffc.high_scale_lib.NonBlockingHashMap$CHM
18: 462641 14804512 java.net.Inet4Address
19: 578801 13891224
java.util.concurrent.ConcurrentSkipListMap$Index
20: 834627 13354032 org.cliffc.high_scale_lib.Counter
21: 556415 13353960 org.apache.cassandra.db.ReadResponse
22: 267483 12839184
org.apache.cassandra.service.ReadCallback
23: 513250 12318000 org.apache.cassandra.net.Message
24: 513248 12317952 org.apache.cassandra.net.Header
25: 501596 12038304
org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
26: 244644 11742912
org.apache.cassandra.db.ThreadSafeSortedColumns
27: 365024 11680768
java.util.concurrent.ConcurrentSkipListMap$HeadIndex
28: 464800 11155200 java.util.Collections$SynchronizedSet
29: 278209 11128360
org.cliffc.high_scale_lib.NonBlockingHashMap
30: 342741 10967712
org.apache.cassandra.utils.ExpiringMap$CacheableObject
31: 199478 9574944 org.apache.cassandra.db.SliceFromReadCommand
32: 278218 8902976
java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
33: 348179 8356296 org.apache.cassandra.db.ColumnFamily
34: 342741 8225784 org.apache.cassandra.net.CallbackInfo
35: 255241 8167712 org.apache.cassandra.db.ArrayBackedSortedColumns
36: 334724 8033376 java.util.ArrayList
37: 499963 7999408 java.util.concurrent.atomic.AtomicReference
38: 319779 7674696 org.apache.cassandra.db.Row
39: 187484 7499360 org.apache.cassandra.db.Memtable$6
40: 464794 7436704 java.util.Hashtable$KeySet
41: 187496 5999872 java.util.TreeMap$KeyIterator
42: 372748 5963968 java.util.concurrent.atomic.AtomicInteger
43: 236877 5685048 org.apache.cassandra.db.filter.QueryPath
44: 236733 5681592 org.apache.cassandra.service.RowDigestResolver
45: 300385 4806160 org.apache.cassandra.utils.SimpleCondition
46: 30726 4245920 * ConstMethodKlass
47: 30726 3696240 * MethodKlass
48: 151726 3641424 org.apache.cassandra.db.SuperColumn
49: 3134 3350536 * ConstantPoolKlass
50: 95622 3059904
java.util.concurrent.locks.ReentrantLock$NonfairSync
51: 45910 2497952 * SymbolKlass
52: 3134 2276040 * InstanceKlassKlass
53: 94544 2269056 java.util.concurrent.LinkedBlockingQueue$Node
54: 93130 2235120
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
55: 133398 2134368
java.util.concurrent.ConcurrentSkipListMap$Values
56: 2813 2099368 * ConstantPoolCacheKlass
57: 37698 1809504 java.util.TreeMap
58: 37783 1511320 java.util.TreeMap$Entry
59: 93602 1497632 java.util.concurrent.locks.ReentrantLock
60: 30768 1476864 java.util.concurrent.LinkedBlockingQueue
61: 85156 1362496
java.util.concurrent.ConcurrentSkipListMap$EntrySet
62: 41451 1326432 org.apache.cassandra.service.RowRepairResolver
63: 41141 1316512 java.util.concurrent.ConcurrentHashMap$HashEntry
64: 32885 1315400
org.apache.cassandra.service.WriteResponseHandler
65: 3110 1277672 * MethodDataKlass
66: 39780 1272960
com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
67: 31556 1262240 org.apache.cassandra.net.AsyncResult
68: 30745 1229800 org.apache.cassandra.db.RangeSliceCommand
69: 37628 1204096 org.apache.cassandra.db.SliceByNamesReadCommand
70: 44528 1068672
com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$WeightedValue
71: 31974 1023168
java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
72: 41435 994440 org.apache.cassandra.service.AsyncRepairCallback
73: 30821 986272 org.apache.cassandra.thrift.SliceRange
74: 30756 984192 java.util.RandomAccessSubList
75: 30745 983840
org.apache.cassandra.service.RangeSliceResponseResolver
76: 29115 931680 org.apache.cassandra.db.DeletedColumn
77: 21676 867040 org.apache.cassandra.db.ExpiringColumn
78: 35339 848136 java.lang.Long
79: 35204 844896 org.apache.cassandra.utils.Pair
80: 30924 742176 java.util.BitSet
81: 30821 739704 org.apache.cassandra.thrift.SlicePredicate
82: 30787 738888 org.apache.cassandra.dht.Bounds
83: 37673 602768 java.util.TreeSet
84: 37654 602464 java.util.TreeMap$KeySet
85: 31598 505568 java.util.concurrent.atomic.AtomicBoolean
86: 6575 420800 java.nio.DirectByteBufferR
87: 2004 416400
java.util.concurrent.ConcurrentHashMap$HashEntry[]
88: 3433 357032 java.lang.Class
89: 8976 287232 java.lang.ref.WeakReference
90: 4999 269008 short[]{code}
> JVM heap is full and GC take no effect ,so many NonBlockingHashMap objects
> in heap
> ------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3537
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3537
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.0
> Environment: 当前堆大小:
> 5,815,955 Kb
> 堆大小的最大值:
> 6,045,696 Kb
> 分配的内存:
> 6,045,696 Kb
> 暂挂结束操作:
> 0 个对象
> 垃圾收集器:
> Name = 'ParNew', Collections = 3,294, Total time spent = 2 minutes
> 垃圾收集器:
> Name = 'ConcurrentMarkSweep', Collections = 5,909, Total time spent = 2 hours
> 17 minutes
>
> 操作系统:
> Linux 2.6.32.12-0.7-default
> 体系结构:
> amd64
> 处理器的数目:
> 16
> 分配的虚拟内存:
> 42,748,416 Kb
> 物理内存总量:
> 24,568,836 Kb
> 可用物理内存:
> 7,136,380 Kb
> 交换空间总量:
> 2,104,472 Kb
> 可用交换空间:
> 1,970,800 Kb
>
> VM 参数:
> -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G
> -Xmn2400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
> -Dcom.sun.management.jmxremote.port=9000
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dpasswd.properties=/opt/obs/cassandra/conf/passwd.properties
> -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties
> -Dlog4j.defaultInitOverride=true
> 类路径:
> /opt/obs/cassandra/conf:/opt/obs/cassandra/build/classes/main:/opt/obs/cassandra/build/classes/thrift:/opt/obs/cassandra/lib/antlr-3.2.jar:/opt/obs/cassandra/lib/apache-cassandra-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-clientutil-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-thrift-1.0.0.jar:/opt/obs/cassandra/lib/avro-1.4.0-fixes.jar:/opt/obs/cassandra/lib/avro-1.4.0-sources-fixes.jar:/opt/obs/cassandra/lib/cassandra_simple_authentication.jar:/opt/obs/cassandra/lib/commons-cli-1.1.jar:/opt/obs/cassandra/lib/commons-codec-1.2.jar:/opt/obs/cassandra/lib/commons-lang-2.4.jar:/opt/obs/cassandra/lib/compress-lzf-0.8.4.jar:/opt/obs/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/obs/cassandra/lib/guava-r08.jar:/opt/obs/cassandra/lib/high-scale-lib-1.1.2.jar:/opt/obs/cassandra/lib/jackson-core-asl-1.4.0.jar:/opt/obs/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/opt/obs/cassandra/lib/jamm-0.2.5.jar:/opt/obs/cassandra/lib/jline-0.9.94.jar:/opt/obs/cassandra/lib/json-simple-1.1.jar:/opt/obs/cassandra/lib/libthrift-0.6.jar:/opt/obs/cassandra/lib/log4j-1.2.16.jar:/opt/obs/cassandra/lib/servlet-api-2.5-20081211.jar:/opt/obs/cassandra/lib/slf4j-api-1.6.1.jar:/opt/obs/cassandra/lib/slf4j-log4j12-1.6.1.jar:/opt/obs/cassandra/lib/snakeyaml-1.6.jar:/opt/obs/cassandra/lib/snappy-java-1.0.3.jar
> Reporter: MaHaiyang
>
> I have 4 cassandra nodes ,and I put about 30G data to db for every nodes .
> It's just 4 days before I start the cluster ,but now every 4 nodes have the
> same problem ,JVM heap is full ,and GC take no effect ,There must be some
> memory leak . Jmap the memory as follow:
> Object Histogram:
> num #instances #bytes Class description
> --------------------------------------------------------------------------
> 1: 15793606 758093088 java.nio.HeapByteBuffer
> 2: 2153811 320138208 java.lang.Object[]
> 3: 6163192 197222144 org.apache.cassandra.db.Column
> 4: 2543836 175890256 int[]
> 5: 2168816 155397192 long[]
> 6: 2078123 116374888
> org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
> 7: 1847111 73884440 java.math.BigInteger
> 8: 1234243 59243664 java.util.Hashtable
> 9: 1770829 58233000 char[]
> 10: 1770627 56660064 java.lang.String
> 11: 1665886 39981264 org.apache.cassandra.db.DecoratedKey
> 12: 692706 38791536
> org.cliffc.high_scale_lib.NonBlockingHashMap$CHM
> 13: 1234274 37172088 java.util.Hashtable$Entry[]
> 14: 1133541 36273312 java.net.Inet4Address
> 15: 738528 35449344
> org.apache.cassandra.service.ReadCallback
> 16: 2078118 33249888 org.cliffc.high_scale_lib.Counter
> 17: 1373886 32973264 org.apache.cassandra.db.ReadResponse
> 18: 1234023 29616552 org.apache.cassandra.net.Message
> 19: 1234019 29616456 org.apache.cassandra.net.Header
> 20: 1846185 29538960 org.apache.cassandra.dht.BigIntegerToken
> 21: 891378 28524096
> org.apache.cassandra.utils.ExpiringMap$CacheableObject
> 22: 692706 27708240
> org.cliffc.high_scale_lib.NonBlockingHashMap
> 23: 1148252 27558048 java.util.Collections$SynchronizedSet
> 24: 541977 26014896
> org.apache.cassandra.db.SliceFromReadCommand
> 25: 998001 23952024
> java.util.concurrent.ConcurrentSkipListMap$Node
> 26: 928792 22291008 java.util.ArrayList
> 27: 692715 22166880
> java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
> 28: 891378 21393072 org.apache.cassandra.net.CallbackInfo
> 29: 1148247 18371952 java.util.Hashtable$KeySet
> 30: 731859 17564616 org.apache.cassandra.db.Row
> 31: 529991 16959712
> org.apache.cassandra.db.ArrayBackedSortedColumns
> 32: 691425 16594200
> org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
> 33: 648580 15565920 org.apache.cassandra.db.filter.QueryPath
> 34: 648338 15560112
> org.apache.cassandra.service.RowDigestResolver
> 35: 971376 15542016
> java.util.concurrent.atomic.AtomicInteger
> 36: 837418 13398688
> org.apache.cassandra.utils.SimpleCondition
> 37: 535614 12854736 org.apache.cassandra.db.ColumnFamily
> 38: 725634 11610144
> java.util.concurrent.atomic.AtomicReference
> 39: 195117 9365616 org.apache.cassandra.db.ThreadSafeSortedColumns
> 40: 281921 9021472
> java.util.concurrent.ConcurrentSkipListMap$HeadIndex
> 41: 277679 8885728
> java.util.concurrent.locks.ReentrantLock$NonfairSync
> 42: 314424 7546176 java.util.concurrent.ConcurrentSkipListMap$Index
> 43: 275186 6604464
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
> 44: 270280 6486720 java.util.concurrent.LinkedBlockingQueue$Node
> 45: 219553 5269272
> org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
> 46: 106436 5108928 java.util.TreeMap
> 47: 122185 4887400 org.apache.cassandra.db.ExpiringColumn
> 48: 189968 4559232 org.apache.cassandra.db.SuperColumn
> 49: 275659 4410544 java.util.concurrent.locks.ReentrantLock
> 50: 90213 4330224 java.util.concurrent.LinkedBlockingQueue
> 51: 107026 4281040 java.util.TreeMap$Entry
> 52: 30501 4222056 * ConstMethodKlass
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
