May I take a look at histograms or jhat for any of the below please Andrew?
What do you see if you get a histogram on any of the OOMEs below (The
'Show heap histogram <http://aa0-000-13.u.powerset.com:7000/histo/>' in
jhat toward end of first page)? Does it look like this?
Class <http://aa0-000-13.u.powerset.com:7000/histo/class> Instance
Count <http://aa0-000-13.u.powerset.com:7000/histo/count> Total Size
<http://aa0-000-13.u.powerset.com:7000/histo/size>
class [B <http://aa0-000-13.u.powerset.com:7000/class/0x7fb09755fc50>
266359 784163182
class org.apache.hadoop.hbase.HStoreKey
<http://aa0-000-13.u.powerset.com:7000/class/0x7fb098371a30> 104113
3331616
class [C <http://aa0-000-13.u.powerset.com:7000/class/0x7fb09755fa28>
23023 2316424
class org.apache.hadoop.hbase.util.SoftValue
<http://aa0-000-13.u.powerset.com:7000/class/0x7fb09840ec48> 45831
2199888
class java.util.HashMap$Entry
<http://aa0-000-13.u.powerset.com:7000/class/0x7fb0975fee00> 48214
1349992
Your query below limiting by 1MB size will not turn up the SoftValue and
HashMap$Entry shown in the above (I think). I'm wondering if your
heaps have any instances these and if so, what are the numbers like?
(If present, see HBASE-900 for current working theory).
Thanks,
St.Ack
Andrew Purtell wrote:
Listing any object class consuming more that 1MB heap:
Class Instance Count Total Size
>From OOME #1:
class [B 218464 1586472174
class java.util.TreeMap$Entry 66290 2717890
class org.apache.hadoop.hbase.HStoreKey 82446 2638272
class [C 27979 2500094
class [J 228 1822792
class [Lorg.apache.hadoop.io.WritableComparable; 222 1822176
>From OOME #2:
class [B 1797666 1509873770
class java.util.TreeMap$Entry 559306 22931546
class org.apache.hadoop.hbase.HStoreKey 682715 21846880
class [J 264 3154840
class [Lorg.apache.hadoop.io.WritableComparable; 250 3004704
class [C 30355 2722664
>From OOME #3:
class [B 1329127 1528647389
class java.util.TreeMap$Entry 407549 16709509
class org.apache.hadoop.hbase.HStoreKey 512322 16394304
class [C 29284 2592440
class [J 234 2585992
class [Lorg.apache.hadoop.io.WritableComparable; 229 2585424
>From OOME #4:
class [B 1928882 1363752468
class java.util.TreeMap$Entry 575415 23592015
class org.apache.hadoop.hbase.HStoreKey 712688 22806016
class [J 359 3937200
class [Lorg.apache.hadoop.io.WritableComparable; 352 3791808
class [C 35421 3264820
I can provide access to the heap dumps or JHAT instances if someone would like.
- Andy
