Please save the jstack of the RS when slow flush is ongoing and confirm whether it stuck at the HDFS writing phase. If so, check log of the local DN co-located with the RegionServer to see whether any notice-able exceptions.
Best Regards, Yu On 23 May 2018 at 14:19, Kang Minwoo <[email protected]> wrote: > @Duo Zhang > This means that you're writing too fast and memstore has reached its upper > limit. Is the flush and compaction fine at RS side? > > -> No, flush took very long time. > I attach code that took a long time to run. (about 30min) > > https://github.com/apache/hbase/blob/branch-1.2/hbase- > server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java# > L2424-L2508 > > Best regards, > Minwoo Kang > ________________________________________ > 보낸 사람: Kang Minwoo <[email protected]> > 보낸 날짜: 2018년 5월 23일 수요일 15:16 > 받는 사람: [email protected] > 제목: Re: can not write to HBase > > I am using salt, prevent write hotspot. > And Table has 4000 regions. > > HBase version is 1.2.6. > > I attach code that took a long time to run. (about 30min) > https://github.com/apache/hbase/blob/branch-1.2/hbase- > server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java# > L2424-L2508 > > Best regards, > Minwoo Kang > > ________________________________________ > 보낸 사람: Anil Gupta <[email protected]> > 보낸 날짜: 2018년 5월 23일 수요일 14:13 > 받는 사람: [email protected] > 제목: Re: can not write to HBase > > It seems you might have a write hotspot. > Are your writes evenly distributed across the cluster? Do you have more > than 15-20 regions for that table? > > Sent from my iPhone > > > On May 22, 2018, at 9:52 PM, Kang Minwoo <[email protected]> > wrote: > > > > I think hbase flush is too slow. > > so memstore reached upper limit. > > > > flush took about 30min. > > I don't know why flush is too long. > > > > Best regards, > > Minwoo Kang > > > > ________________________________________ > > 보낸 사람: 张铎(Duo Zhang) <[email protected]> > > 보낸 날짜: 2018년 5월 23일 수요일 11:37 > > 받는 사람: hbase-user > > 제목: Re: can not write to HBase > > > > org.apache.hadoop.hbase.RegionTooBusyException: > > org.apache.hadoop.hbase.RegionTooBusyException: > > Above memstore limit, regionName={region}, server={server}, > > memstoreSize=2600502128, blockingMemStoreSize=2600468480 > > > > This means that you're writing too fast and memstore has reached its > upper > > limit. Is the flush and compaction fine at RS side? > > > > 2018-05-23 10:20 GMT+08:00 Kang Minwoo <[email protected]>: > > > >> attach client exception and stacktrace. > >> > >> I've looked more. > >> It seems to be the reason why it takes 1290 seconds to flush in > the Region > >> Server. > >> > >> 2018-05-23T07:24:31.202 [INFO] Call exception, tries=34, > retries=35, > >> started=513393 ms ago, cancelled=false, msg=row '{row}' on table > '{table}' > >> at region={region}, hostname={host}, seqNum=155455658 > >> 2018-05-23T07:24:31.208 [ERROR] > >> java.lang.RuntimeException: com.google.protobuf.ServiceException: > Error > >> calling method MultiRowMutationService.MutateRows > >> at com.google.common.base.Throwables.propagate( > Throwables.java:160) > >> ~[stormjar.jar:?] > >> at ... > >> at org.apache.storm.daemon.executor$fn__8058$tuple_ > >> action_fn__8060.invoke(executor.clj:731) > [storm-core-1.0.2.jar:1.0.2] > >> at org.apache.storm.daemon.executor$mk_task_receiver$fn__ > 7979.invoke(executor.clj:464) > >> [storm-core-1.0.2.jar:1.0.2] > >> at org.apache.storm.disruptor$clojure_handler$reify__7492. > onEvent(disruptor.clj:40) > >> [storm-core-1.0.2.jar:1.0.2] > >> at org.apache.storm.utils.DisruptorQueue. > consumeBatchToCursor(DisruptorQueue.java:451) > >> [storm-core-1.0.2.jar:1.0.2] > >> at org.apache.storm.utils.DisruptorQueue. > >> consumeBatchWhenAvailable(DisruptorQueue.java:430) > >> [storm-core-1.0.2.jar:1.0.2] > >> at org.apache.storm.disruptor$ > consume_batch_when_available.invoke(disruptor.clj:73) > >> [storm-core-1.0.2.jar:1.0.2] > >> at org.apache.storm.daemon.executor$fn__8058$fn__8071$fn_ > _8124.invoke(executor.clj:850) > >> [storm-core-1.0.2.jar:1.0.2] > >> at org.apache.storm.util$async_ > loop$fn__624.invoke(util.clj:484) > >> [storm-core-1.0.2.jar:1.0.2] > >> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > >> at java.lang.Thread.run(Thread.java:745) [?:1.7.0_80] > >> Caused by: com.google.protobuf.ServiceException: Error calling > method > >> MultiRowMutationService.MutateRows > >> at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel. > >> callBlockingMethod(CoprocessorRpcChannel.java:75) > ~[stormjar.jar:?] > >> at org.apache.hadoop.hbase.protobuf.generated. > >> MultiRowMutationProtos$MultiRowMutationService$ > BlockingStub.mutateRows( > >> MultiRowMutationProtos.java:2149) ~[stormjar.jar:?] > >> at ... > >> ... 13 more > >> Caused by: org.apache.hadoop.hbase.client. > RetriesExhaustedException: > >> Failed after attempts=35, exceptions: > >> Wed May 23 07:15:57 KST 2018, RpcRetryingCaller{ > globalStartTime=1527027357808, > >> pause=100, retries=35}, org.apache.hadoop.hbase. > RegionTooBusyException: > >> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore > limit, > >> regionName={region}, server={server}, memstoreSize=2600502128, > >> blockingMemStoreSize=2600468480 > >> at org.apache.hadoop.hbase.regionserver.HRegion. > >> checkResources(HRegion.java:3649) > >> at org.apache.hadoop.hbase.regionserver.HRegion. > >> processRowsWithLocks(HRegion.java:6935) > >> at org.apache.hadoop.hbase.regionserver.HRegion. > >> mutateRowsWithLocks(HRegion.java:6885) > >> at org.apache.hadoop.hbase.coprocessor. > MultiRowMutationEndpoint. > >> mutateRows(MultiRowMutationEndpoint.java:116) > >> at org.apache.hadoop.hbase.protobuf.generated. > >> MultiRowMutationProtos$MultiRowMutationService.callMethod( > >> MultiRowMutationProtos.java:2053) > >> at org.apache.hadoop.hbase.regionserver.HRegion. > >> execService(HRegion.java:7875) > >> at org.apache.hadoop.hbase.regionserver.RSRpcServices. > >> execServiceOnRegion(RSRpcServices.java:2008) > >> > >> > >> Best regards, > >> Minwoo Kang > >> > >> ________________________________________ > >> 보낸 사람: 张铎(Duo Zhang) <[email protected]> > >> 보낸 날짜: 2018년 5월 23일 수요일 09:22 > >> 받는 사람: hbase-user > >> 제목: Re: can not write to HBase > >> > >> What is the exception? And the stacktrace? > >> > >> 2018-05-23 8:17 GMT+08:00 Kang Minwoo <[email protected]>: > >> > >>> Hello, Users > >>> > >>> My HBase client does not work after print the following logs. > >>> > >>> Call exception, tries=23, retries=35, started=291277 ms ago, > >>> cancelled=false, msg=row '{row}' on table '{table}' at > region={region}, > >>> hostname={hostname}, seqNum=100353531 > >>> > >>> There are no special logs in the Master and Region Servers. > >>> Is it something wrong in the client? > >>> > >>> Best regards, > >>> Minwoo Kang > >>> > >> > </[email protected]></[email protected]></minwoo. > [email protected]></[email protected]></[email protected]></ > [email protected]></[email protected]>
