Sorry, I have a mistake. I put wrong HBase Configuration. So that error occurs.
My client's server located in a different datacenter with our HBase server. I checked the network latency. The servers' network latency in the same data center is under 0.1ms. But servers' network latency in other data centers is > 5ms. I think to create an HBase connection takes more time. After creating an HBase connection, Scan data speed difference is little. Best regards, Minwoo Kang ________________________________________ 보낸 사람: Josh Elser <els...@apache.org> 보낸 날짜: 2019년 2월 28일 목요일 00:15 받는 사람: user@hbase.apache.org 제목: Re: HBase client spent most time in ThreadPoolExecutor Do you have any regions in transition? Does HBCK report any problems? It sounds to me that a client is stuck polling meta to look for the location of a Region which it cannot find for some reason. Finding the location of a Region from meta should not take more than a second. On 2/27/19 12:34 AM, Kang Minwoo wrote: > MetaScan is so slow. > When I invoked `regionLocator.getAllRegionLocations()` method, It throw > `org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the > locations` Exception. > > Best regards, > Minwoo Kang > > ________________________________________ > 보낸 사람: Kang Minwoo <minwoo.k...@outlook.com> > 보낸 날짜: 2019년 2월 27일 수요일 13:48 > 받는 사람: user@hbase.apache.org > 제목: Re: HBase client spent most time in ThreadPoolExecutor > > Thank you for your reply. > > I looked around thread dumps. > > My Client has 256 connections. > but only one connection state is RUNNABLE. others state are TIMED_WAITING > (parking). > > Best regards, > Minwoo Kang > > ________________________________________ > 보낸 사람: Josh Elser <els...@apache.org> > 보낸 날짜: 2019년 2월 27일 수요일 01:32 > 받는 사람: user@hbase.apache.org > 제목: Re: HBase client spent most time in ThreadPoolExecutor > > Minwoo, > > You have found an idle thread in the threadpool that is waiting for > work. This is not the source of your slowness. The thread is polling the > internal queue of work, waiting for the next "unit" of something to do. > > You should include threads like these from your analysis. > > On 2/26/19 8:32 AM, Kang Minwoo wrote: >> Hello Users, >> >> I have a question. >> >> My client complains to me, HBase scan spent too much time. >> So I started to debug. >> >> I profiled the HBase Client Application using hprof. >> The App spent the most time in below stack trace. >> >> ---- >> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:Unknown >> line) >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:Unknown >> line) >> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:Unknown >> line) >> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:Unknown >> line) >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:Unknown >> line) >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:Unknown >> line) >> java.lang.Thread.run(Thread.java:Unknown line) >> --- >> >> --- >> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:Unknown >> line) >> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:Unknown >> line) >> --- >> >> they spent 92% of total time. >> >> I don't understand why they spent a lot of time. >> Do you have any idea? >> >> Best regards, >> Minwoo Kang >>
