What LZO version are you using? You aren't running out of regular heap, you are running out of "Direct buffer memory" which is capped to prevent mishaps. There is a flag to increase that size:
-XX:MaxDirectMemorySize=100m etc enjoy, -ryan On Thu, Dec 16, 2010 at 3:07 PM, Sandy Pratt <[email protected]> wrote: > Hello HBasers, > > I had a regionserver crash recently, and in perusing the logs it looks like > it simply had a bit too little memory. I'm running with 2200 MB heap on > reach regionserver. I plan to shave a bit off the child VM allowance in > favor of the regionserver to correct this, probably bringing it up to 2500 > MB. My question is if there is any more specific memory allocation I should > make rather than simply giving more to the RS. I wonder about this because > of the following: > > load=(requests=0, regions=709, usedHeap=1349, maxHeap=2198) > > which suggests to me that there was heap available, but the RS couldn't use > it for some reason. > > Conjecture: I do run with LZO compression, so I wonder if I could be hitting > that memory leak referenced earlier on the list. I know there's a new > version of the LZO library available that I should upgrade to, but is it also > possible to simply alter the table to gzip compression and do a major > compaction, then uninstall LZO once that completes? > > Log follows: > > 2010-12-15 20:01:05,239 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region > ets.events,36345112f5654a29b308014f89c108e6,12798158203 > 11.1063152548 > 2010-12-15 20:01:05,239 DEBUG org.apache.hadoop.hbase.regionserver.Store: > Major compaction triggered on store f1; time since last major compaction > 119928149ms > 2010-12-15 20:01:05,240 INFO org.apache.hadoop.hbase.regionserver.Store: > Started compaction of 2 file(s) in f1 of > ets.events,36345112f5654a29b308014f89c108e6,12 > 79815820311.1063152548 into > hdfs://ets-lax-prod-hadoop-01.corp.adobe.com:54310/hbase/ets.events/1063152548/.tmp, > sequenceid=25718885315 > 2010-12-15 20:01:19,403 WARN org.apache.hadoop.hbase.regionserver.Store: Not > in setorg.apache.hadoop.hbase.regionserver.storescan...@7466c84 > 2010-12-15 20:01:19,572 FATAL > org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server > serverName=ets-lax-prod-hadoop-02.corp.adobe.com,60020, > 1289682554219, load=(requests=0, regions=709, usedHeap=1349, maxHeap=2198): > Uncaught exception in service thread regionserver60020.compactor > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:656) > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:113) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305) > at > com.hadoop.compression.lzo.LzoCompressor.init(LzoCompressor.java:223) > at > com.hadoop.compression.lzo.LzoCompressor.reinit(LzoCompressor.java:207) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:105) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112) > at > org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:198) > at > org.apache.hadoop.hbase.io.hfile.HFile$Writer.getCompressingStream(HFile.java:391) > at > org.apache.hadoop.hbase.io.hfile.HFile$Writer.newBlock(HFile.java:377) > at > org.apache.hadoop.hbase.io.hfile.HFile$Writer.checkBlockBoundary(HFile.java:348) > at org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:530) > at org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:495) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:817) > at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:811) > at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:670) > at > org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:722) > at > org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:671) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:84) > 2010-12-15 20:01:19,586 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: > request=0.0, regions=709, stores=709, storefiles=731, storefileIndexSize=418, > memstoreSize=33, compactionQueueSize=15, usedHeap=856, maxHeap=2198, > blockCacheSize=366779472, blockCacheFree=87883088, blockCacheCount=5494, > blockCacheHitRatio=0 > 2010-12-15 20:01:20,571 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > server on 60020 > > Thanks, > > Sandy > >
