Thank you for reply.
I found one handler thread state is RUNNABLE.
Other handler thread state is TIMED_WAITING.
I think the RUNNABLE handler thread is issue.
----
[Thread dump]
"RpcServer"
java.lang.Thread.State: RUNNABLE
at
org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:2020)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:208)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:184)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:174)
at
java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:721)
at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
at java.util.PriorityQueue.poll(PriorityQueue.java:595)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:355)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:123)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5733)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5896)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5670)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2580)
- locked <> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33648)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
----
I don't know why that handler running is running too long.
I will be more research on how to avoid this problem.
Best regards,
Minwoo Kang
________________________________________
보낸 사람: Allan Yang <allan...@apache.org>
보낸 날짜: 2018년 7월 11일 수요일 18:54
받는 사람: user@hbase.apache.org
제목: Re: Region state is PENDING_CLOSE persists.
There must be a handler thread is running (or stuck) somewhere, so the
close region thread can't obtain the write lock. You can look closely in
your thread dump.
The handler thread you pasted above it is just a thread can't obtain the
read lock since the close thread is trying write lock.
Best Regards
Allan Yang
Kang Minwoo <minwoo.k...@outlook.com> 于2018年7月11日周三 下午2:25写道：
> Hello.
>
> Occasionally, when closing a region, the RS_CLOSE_REGION thread is unable
> to acquire a lock and is still in the WAITING.
> (These days, the cluster load increase.)
> So the Region state is PENDING_CLOSE persists.
> The thread holding the lock is the RPC handler.
>
> If you have any good tips on moving regions, please share them.
> It would be nice if the timeout could be set.
>
> The HBase version is 1.2.6.
>
> Best regards,
> Minwoo Kang
>
> ----
>
> [thread dump]
> "RS_CLOSE_REGION" waiting on condition [abc]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <abcd> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1426)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1372)
> - locked <e> (a java.lang.Object)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:138)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <f> (a java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "RpcServer.handler" waiting on condition [bcd]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <abcd> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8177)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8164)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8073)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2547)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2541)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6830)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6809)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2049)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
>
>
>
