Do you have any regions in transition? Does HBCK report any problems?
It sounds to me that a client is stuck polling meta to look for the
location of a Region which it cannot find for some reason. Finding the
location of a Region from meta should not take more than a second.
On 2/27/19 12:34 AM, Kang Minwoo wrote:
MetaScan is so slow.
When I invoked `regionLocator.getAllRegionLocations()` method, It throw
`org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the
locations` Exception.
Best regards,
Minwoo Kang
________________________________________
보낸 사람: Kang Minwoo <[email protected]>
보낸 날짜: 2019년 2월 27일 수요일 13:48
받는 사람: [email protected]
제목: Re: HBase client spent most time in ThreadPoolExecutor
Thank you for your reply.
I looked around thread dumps.
My Client has 256 connections.
but only one connection state is RUNNABLE. others state are TIMED_WAITING
(parking).
Best regards,
Minwoo Kang
________________________________________
보낸 사람: Josh Elser <[email protected]>
보낸 날짜: 2019년 2월 27일 수요일 01:32
받는 사람: [email protected]
제목: Re: HBase client spent most time in ThreadPoolExecutor
Minwoo,
You have found an idle thread in the threadpool that is waiting for
work. This is not the source of your slowness. The thread is polling the
internal queue of work, waiting for the next "unit" of something to do.
You should include threads like these from your analysis.
On 2/26/19 8:32 AM, Kang Minwoo wrote:
Hello Users,
I have a question.
My client complains to me, HBase scan spent too much time.
So I started to debug.
I profiled the HBase Client Application using hprof.
The App spent the most time in below stack trace.
----
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:Unknown line)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:Unknown
line)
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:Unknown
line)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:Unknown
line)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:Unknown
line)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:Unknown
line)
java.lang.Thread.run(Thread.java:Unknown line)
---
---
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:Unknown
line)
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:Unknown
line)
---
they spent 92% of total time.
I don't understand why they spent a lot of time.
Do you have any idea?
Best regards,
Minwoo Kang
