Yes, we are on 64-bits, it's 2010 right? :P J-D
On Wed, Jun 9, 2010 at 11:34 AM, Vidhyashankar Venkataraman <vidhy...@yahoo-inc.com> wrote: >>> You wrote you have 24GB available, why only give 3 to HBase? You don't like >>> him? > Hehe.. > So, are you running 64-bit Java then? I thought java-32 doesn't allow more > than around 3-4 gigs of RAM.. > > On 6/9/10 11:26 AM, "Jean-Daniel Cryans" <jdcry...@apache.org> wrote: > > On Wed, Jun 9, 2010 at 11:16 AM, Vidhyashankar Venkataraman > <vidhy...@yahoo-inc.com> wrote: >> What do you mean by pastebinning it? I will try hosting it on a webserver.. > > pastebin.com > >> >> I know that OOME is Java running out of heap space: Can you let me know what >> are the usual causes for OOME happening in Hbase? Was I pounding the servers >> a bit too hard with updates? > > Well you know why it happens then ;) > > So, since you sent me the log in private until you get it somewhere > public, I see this very important line: > > 2010-06-09 02:31:23,874 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: > request=0.0, regions=467, stores=933, storefiles=3272, > storefileIndexSize=0, memstoreSize=1181, compactionQueueSize=59, > usedHeap=2796, maxHeap=2999, blockCacheSize=681651688, > blockCacheFree=261889816, blockCacheCount=7, blockCacheHitRatio=61, > fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0 > > This is outputted by the region servers when they shut down. As you > can see, the used heap is 2796 over 2999, and then the OOME was > triggered during: > > 2010-06-09 02:31:23,874 FATAL > org.apache.hadoop.hbase.regionserver.HRegionServer: Set stop flag in > regionserver/74.6.71.59:60020.compactor > java.lang.OutOfMemoryError: Java heap space > at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39) > at java.nio.ByteBuffer.allocate(ByteBuffer.java:312) > at > org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1019) > at > org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:971) > at > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1163) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79) > at > org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingStoreScanner.java:123) > > Since your max file size is 2GB, which btw is fairly big compared to > your actual available heap, I'd say your settings are set way too high > for what you gave to HBase to play with. You wrote you have 24GB > available, why only give 3 to HBase? You don't like him? > > Jokes apart, HBase is a database, database needs memory, etc. We have > the exact same HW here, and we give 8GB to HBase. Try also setting > compressed object pointers on the JVM, passing it through hbase-env.sh > > J-D > >
