[jira] [Comment Edited] (HBASE-14178) regionserver blocks because of waiting for offsetLock
[
https://issues.apache.org/jira/browse/HBASE-14178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14659574#comment-14659574
]
Duo Zhang edited comment on HBASE-14178 at 8/6/15 6:46 AM:
---
[~ram_krish] A bit strange but now, {{family.isBlockCacheEnabled}} only means
{{cacheDataOnRead}} ... {{cacheDataOnWrite}} and some other configurations such
as {{prefetchOnOpen}} are separated which means you can set them to {{true}}
and will take effect even if you explicitly call
{{family.setBlockCacheEnabled(false)}}. So theoretically even if you disable BC
at family level, it is still possible that we could find the block in BC...
was (Author: apache9):
[~ram_krish] A bit strange but now, {{family.isBlockCacheEnabled}} only means
{{cacheDataOnRead }}... {{cacheDataOnWrite}} and some other configurations such
as {{prefetchOnOpen}} are separated which means you can set them to {{true}}
and will take effect even if you explicitly call
{{family.setBlockCacheEnabled(false)}}. So theoretically even if you disable BC
at family level, it is still possible that we could find the block in BC...
regionserver blocks because of waiting for offsetLock
-
Key: HBASE-14178
URL: https://issues.apache.org/jira/browse/HBASE-14178
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.6
Reporter: Heng Chen
Priority: Critical
Fix For: 0.98.6
Attachments: HBASE-14178-0.98.patch, HBASE-14178.patch,
HBASE-14178_v1.patch, HBASE-14178_v2.patch, HBASE-14178_v3.patch,
HBASE-14178_v4.patch, HBASE-14178_v5.patch, HBASE-14178_v6.patch, jstack
My regionserver blocks, and all client rpc timeout.
I print the regionserver's jstack, it seems a lot of threads were blocked
for waiting offsetLock, detail infomation belows:
PS: my table's block cache is off
{code}
B.DefaultRpcServer.handler=2,queue=2,port=60020 #82 daemon prio=5 os_prio=0
tid=0x01827000 nid=0x2cdc in Object.wait() [0x7f3831b72000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.hadoop.hbase.util.IdLock.getLockEntry(IdLock.java:79)
- locked 0x000773af7c18 (a
org.apache.hadoop.hbase.util.IdLock$Entry)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:352)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:253)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:524)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:572)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:257)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:173)
at
org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:313)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:269)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:695)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:683)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:533)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:140)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3889)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3969)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3847)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3820)
- locked 0x0005e5c55ad0 (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3807)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4779)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4753)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2916)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29583)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
[jira] [Comment Edited] (HBASE-14178) regionserver blocks because of waiting for offsetLock
[
https://issues.apache.org/jira/browse/HBASE-14178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14659728#comment-14659728
]
Duo Zhang edited comment on HBASE-14178 at 8/6/15 9:23 AM:
---
Pushed to all branches.
Thanks [~chenheng] for the patch(and the quick turn around at last although I
have done some revert commits...).
Thanks all guys who helped review the patch.
was (Author: apache9):
Thanks [~chenheng] for the patch(and the quick turn around at last although I
have done some revert commits...).
Thanks all guys who helped review the patch.
regionserver blocks because of waiting for offsetLock
-
Key: HBASE-14178
URL: https://issues.apache.org/jira/browse/HBASE-14178
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.6
Reporter: Heng Chen
Assignee: Heng Chen
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2
Attachments: HBASE-14178-0.98.patch, HBASE-14178-0.98_v8.patch,
HBASE-14178-branch_1_v8.patch, HBASE-14178.patch, HBASE-14178_v1.patch,
HBASE-14178_v2.patch, HBASE-14178_v3.patch, HBASE-14178_v4.patch,
HBASE-14178_v5.patch, HBASE-14178_v6.patch, HBASE-14178_v7.patch,
HBASE-14178_v8.patch, jstack
My regionserver blocks, and all client rpc timeout.
I print the regionserver's jstack, it seems a lot of threads were blocked
for waiting offsetLock, detail infomation belows:
PS: my table's block cache is off
{code}
B.DefaultRpcServer.handler=2,queue=2,port=60020 #82 daemon prio=5 os_prio=0
tid=0x01827000 nid=0x2cdc in Object.wait() [0x7f3831b72000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.hadoop.hbase.util.IdLock.getLockEntry(IdLock.java:79)
- locked 0x000773af7c18 (a
org.apache.hadoop.hbase.util.IdLock$Entry)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:352)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:253)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:524)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:572)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:257)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:173)
at
org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:313)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:269)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:695)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:683)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:533)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:140)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3889)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3969)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3847)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3820)
- locked 0x0005e5c55ad0 (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3807)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4779)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4753)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2916)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29583)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- 0x0005e5c55c08 (a
