[
https://issues.apache.org/jira/browse/HBASE-10500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898087#comment-13898087
]
Nick Dimiduk commented on HBASE-10500:
--------------------------------------
Here's the full stack trace:
{noformat}
Exception in thread "main" java.io.IOException: java.lang.OutOfMemoryError:
Direct buffer memory
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:731)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:638)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:609)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:595)
at
org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4195)
at
org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4154)
at
org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4127)
at
org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4205)
at
org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4085)
at
org.apache.hadoop.hbase.util.HBaseFsckRepair.createHDFSRegionDir(HBaseFsckRepair.java:190)
at
org.apache.hadoop.hbase.util.HBaseFsck$TableInfo$HDFSIntegrityFixer.handleHoleInRegionChain(HBaseFsck.java:2312)
at
org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.checkRegionChain(HBaseFsck.java:2492)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkHdfsIntegrity(HBaseFsck.java:1226)
at
org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:741)
at
org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:386)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:475)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4029)
at
org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:3838)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3826)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at
org.apache.hadoop.hbase.util.ByteBufferArray.<init>(ByteBufferArray.java:65)
at
org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.<init>(ByteBufferIOEngine.java:44)
at
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:270)
at
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:210)
at
org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:399)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:143)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:231)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3309)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:702)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:699)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
{noformat}
> hbck and OOM when BucketCache is enabled
> ----------------------------------------
>
> Key: HBASE-10500
> URL: https://issues.apache.org/jira/browse/HBASE-10500
> Project: HBase
> Issue Type: Bug
> Components: hbck
> Affects Versions: 0.98.0
> Reporter: Nick Dimiduk
> Assignee: Nick Dimiduk
>
> Running {{hbck --repair}} when BucketCache is enabled in offheap mode can
> cause OOM. This is apparently because {{bin/hbase}} does not include
> $HBASE_REGIONSERVER_OPTS for hbck. It instantiates an HRegion instance as
> part of HDFSIntegrityFixer.handleHoleInRegionChain. That HRegion initializes
> its CacheConfig, which doesn't have the necessary Direct Memory.
> Possible solutions include:
> - disable blockcache in the config used by hbck when running its repairs
> - include HBASE_REGIONSERVER_OPTS in the HBaseFSCK startup arguments
> I'm leaning toward the former because it's possible that hbck is run on a
> host with the same hardware profile as the RS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
