>> You wrote you have 24GB available, why only give 3 to HBase? You don't like
>> him?
Hehe..
So, are you running 64-bit Java then? I thought java-32 doesn't allow more than
around 3-4 gigs of RAM..
On 6/9/10 11:26 AM, "Jean-Daniel Cryans" <jdcry...@apache.org> wrote:
On Wed, Jun 9, 2010 at 11:16 AM, Vidhyashankar Venkataraman
<vidhy...@yahoo-inc.com> wrote:
> What do you mean by pastebinning it? I will try hosting it on a webserver..
pastebin.com
>
> I know that OOME is Java running out of heap space: Can you let me know what
> are the usual causes for OOME happening in Hbase? Was I pounding the servers
> a bit too hard with updates?
Well you know why it happens then ;)
So, since you sent me the log in private until you get it somewhere
public, I see this very important line:
2010-06-09 02:31:23,874 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=467, stores=933, storefiles=3272,
storefileIndexSize=0, memstoreSize=1181, compactionQueueSize=59,
usedHeap=2796, maxHeap=2999, blockCacheSize=681651688,
blockCacheFree=261889816, blockCacheCount=7, blockCacheHitRatio=61,
fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
This is outputted by the region servers when they shut down. As you
can see, the used heap is 2796 over 2999, and then the OOME was
triggered during:
2010-06-09 02:31:23,874 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: Set stop flag in
regionserver/74.6.71.59:60020.compactor
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1019)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:971)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1163)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at
org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingStoreScanner.java:123)
Since your max file size is 2GB, which btw is fairly big compared to
your actual available heap, I'd say your settings are set way too high
for what you gave to HBase to play with. You wrote you have 24GB
available, why only give 3 to HBase? You don't like him?
Jokes apart, HBase is a database, database needs memory, etc. We have
the exact same HW here, and we give 8GB to HBase. Try also setting
compressed object pointers on the JVM, passing it through hbase-env.sh
J-D
