Yes, I use reverse scan at that time. The situation you have shared exactly matches our situation.
Thank you for share Good material !! And I think HBASE-15871 should backport 1.2 branch. ---- (Yes, I am Korean.) Best regards, Minwoo Kang ________________________________________ 보낸 사람: Jungdae Kim <kjd9...@gmail.com> 보낸 날짜: 2018년 5월 29일 화요일 11:21 받는 사람: user@hbase.apache.org 제목: Re: can not write to HBase Memstore flusher waits until all current store scanners' operations(next()) are done. I think you have to check why scaner.next() is too slow. Does your applications(hbase client) use reverse scans? If so, i think you are sufferiing the issue related to HBASE-15871( https://issues.apache.org/jira/browse/HBASE-15871) If you want more details about this issue or a memstore flusher, please check the following link, notes written to Korean.(case #4: page 48 ~ 55) (I think you are Korean. ^^) - https://www.evernote.com/shard/s167/sh/39eb6b44-25e7-4e61-ad2a-a0d1b076c7d1/159db49e3e49b189 Best regards, Jeongdae Kim 김정대 드림. On Thu, May 24, 2018 at 1:22 PM, Kang Minwoo <minwoo.k...@outlook.com> wrote: > I have a same error on today. > thread dump is here. > > ---------------- > > Thread 286 (MemStoreFlusher.1): > State: WAITING > Blocked count: 10704 > Waited count: 10936 > Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@2afc16fd > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > java.util.concurrent.locks.AbstractQueuedSynchronizer. > parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued( > AbstractQueuedSynchronizer.java:870) > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire( > AbstractQueuedSynchronizer.java:1199) > java.util.concurrent.locks.ReentrantLock$NonfairSync. > lock(ReentrantLock.java:209) > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) > org.apache.hadoop.hbase.regionserver.StoreScanner. > updateReaders(StoreScanner.java:693) > org.apache.hadoop.hbase.regionserver.HStore. > notifyChangedReadersObservers(HStore.java:1093) > org.apache.hadoop.hbase.regionserver.HStore. > updateStorefiles(HStore.java:1072) > org.apache.hadoop.hbase.regionserver.HStore.access$ > 700(HStore.java:118) > org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit( > HStore.java:2310) > org.apache.hadoop.hbase.regionserver.HRegion. > internalFlushCacheAndCommit(HRegion.java:2386) > org.apache.hadoop.hbase.regionserver.HRegion. > internalFlushcache(HRegion.java:2108) > org.apache.hadoop.hbase.regionserver.HRegion. > internalFlushcache(HRegion.java:2070) > org.apache.hadoop.hbase.regionserver.HRegion. > flushcache(HRegion.java:1961) > org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1887) > org.apache.hadoop.hbase.regionserver.MemStoreFlusher. > flushRegion(MemStoreFlusher.java:514) > org.apache.hadoop.hbase.regionserver.MemStoreFlusher. > flushRegion(MemStoreFlusher.java:475) > org.apache.hadoop.hbase.regionserver.MemStoreFlusher. > access$900(MemStoreFlusher.java:75) > > > ---------------- > > I deleted many row these days. > I think that affected. > > Best regards, > Minwoo Kang > > ________________________________________ > 보낸 사람: Kang Minwoo <minwoo.k...@outlook.com> > 보낸 날짜: 2018년 5월 23일 수요일 16:53 > 받는 사람: Hbase-User > 제목: Re: can not write to HBase > > In HRegion#internalFlushCacheAndCommit > There is following code. > > synchronized (this) { > notifyAll(); // FindBugs NN_NAKED_NOTIFY > } > > one question. > Where is the lock acquired? > > Best regards, > Minwoo Kang > > ________________________________________ > 보낸 사람: Kang Minwoo <minwoo.k...@outlook.com> > 보낸 날짜: 2018년 5월 23일 수요일 16:37 > 받는 사람: Hbase-User > 제목: Re: can not write to HBase > > Next time, If I have a same problem. I will save the jstack of the RS. > (I could not think of saving jstack this time.) > > I did not see any special logs. > There was only a warn log that the HBase Scan was slow. > > Best regards, > Minwoo Kang > > ________________________________________ > 보낸 사람: Yu Li <car...@gmail.com> > 보낸 날짜: 2018년 5월 23일 수요일 15:53 > 받는 사람: Hbase-User > 제목: Re: can not write to HBase > > Please save the jstack of the RS when slow flush is ongoing and confirm > whether it stuck at the HDFS writing phase. If so, check log of the local > DN co-located with the RegionServer to see whether any notice-able > exceptions. > > Best Regards, > Yu > > On 23 May 2018 at 14:19, Kang Minwoo <minwoo.k...@outlook.com> wrote: > > > @Duo Zhang > > This means that you're writing too fast and memstore has reached its > upper > > limit. Is the flush and compaction fine at RS side? > > > > -> No, flush took very long time. > > I attach code that took a long time to run. (about 30min) > > > > https://github.com/apache/hbase/blob/branch-1.2/hbase- > > server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java# > > L2424-L2508 > > > > Best regards, > > Minwoo Kang > > ________________________________________ > > 보낸 사람: Kang Minwoo <minwoo.k...@outlook.com> > > 보낸 날짜: 2018년 5월 23일 수요일 15:16 > > 받는 사람: user@hbase.apache.org > > 제목: Re: can not write to HBase > > > > I am using salt, prevent write hotspot. > > And Table has 4000 regions. > > > > HBase version is 1.2.6. > > > > I attach code that took a long time to run. (about 30min) > > https://github.com/apache/hbase/blob/branch-1.2/hbase- > > server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java# > > L2424-L2508 > > > > Best regards, > > Minwoo Kang > > > > ________________________________________ > > 보낸 사람: Anil Gupta <anilgupt...@gmail.com> > > 보낸 날짜: 2018년 5월 23일 수요일 14:13 > > 받는 사람: user@hbase.apache.org > > 제목: Re: can not write to HBase > > > > It seems you might have a write hotspot. > > Are your writes evenly distributed across the cluster? Do you have more > > than 15-20 regions for that table? > > > > Sent from my iPhone > > > > > On May 22, 2018, at 9:52 PM, Kang Minwoo <minwoo.k...@outlook.com> > > wrote: > > > > > > I think hbase flush is too slow. > > > so memstore reached upper limit. > > > > > > flush took about 30min. > > > I don't know why flush is too long. > > > > > > Best regards, > > > Minwoo Kang > > > > > > ________________________________________ > > > 보낸 사람: 张铎(Duo Zhang) <palomino...@gmail.com> > > > 보낸 날짜: 2018년 5월 23일 수요일 11:37 > > > 받는 사람: hbase-user > > > 제목: Re: can not write to HBase > > > > > > org.apache.hadoop.hbase.RegionTooBusyException: > > > org.apache.hadoop.hbase.RegionTooBusyException: > > > Above memstore limit, regionName={region}, server={server}, > > > memstoreSize=2600502128, blockingMemStoreSize=2600468480 > > > > > > This means that you're writing too fast and memstore has reached its > > upper > > > limit. Is the flush and compaction fine at RS side? > > > > > > 2018-05-23 10:20 GMT+08:00 Kang Minwoo <minwoo.k...@outlook.com>: > > > > > >> attach client exception and stacktrace. > > >> > > >> I've looked more. > > >> It seems to be the reason why it takes 1290 seconds to flush in > > the Region > > >> Server. > > >> > > >> 2018-05-23T07:24:31.202 [INFO] Call exception, tries=34, > > retries=35, > > >> started=513393 ms ago, cancelled=false, msg=row '{row}' on table > > '{table}' > > >> at region={region}, hostname={host}, seqNum=155455658 > > >> 2018-05-23T07:24:31.208 [ERROR] > > >> java.lang.RuntimeException: com.google.protobuf. > ServiceException: > > Error > > >> calling method MultiRowMutationService.MutateRows > > >> at com.google.common.base.Throwables.propagate( > > Throwables.java:160) > > >> ~[stormjar.jar:?] > > >> at ... > > >> at org.apache.storm.daemon.executor$fn__8058$tuple_ > > >> action_fn__8060.invoke(executor.clj:731) > > [storm-core-1.0.2.jar:1.0.2] > > >> at org.apache.storm.daemon. > executor$mk_task_receiver$fn__ > > 7979.invoke(executor.clj:464) > > >> [storm-core-1.0.2.jar:1.0.2] > > >> at org.apache.storm.disruptor$ > clojure_handler$reify__7492. > > onEvent(disruptor.clj:40) > > >> [storm-core-1.0.2.jar:1.0.2] > > >> at org.apache.storm.utils.DisruptorQueue. > > consumeBatchToCursor(DisruptorQueue.java:451) > > >> [storm-core-1.0.2.jar:1.0.2] > > >> at org.apache.storm.utils.DisruptorQueue. > > >> consumeBatchWhenAvailable(DisruptorQueue.java:430) > > >> [storm-core-1.0.2.jar:1.0.2] > > >> at org.apache.storm.disruptor$ > > consume_batch_when_available.invoke(disruptor.clj:73) > > >> [storm-core-1.0.2.jar:1.0.2] > > >> at org.apache.storm.daemon. > executor$fn__8058$fn__8071$fn_ > > _8124.invoke(executor.clj:850) > > >> [storm-core-1.0.2.jar:1.0.2] > > >> at org.apache.storm.util$async_ > > loop$fn__624.invoke(util.clj:484) > > >> [storm-core-1.0.2.jar:1.0.2] > > >> at clojure.lang.AFn.run(AFn.java:22) > [clojure-1.7.0.jar:?] > > >> at java.lang.Thread.run(Thread.java:745) [?:1.7.0_80] > > >> Caused by: com.google.protobuf.ServiceException: Error calling > > method > > >> MultiRowMutationService.MutateRows > > >> at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel. > > >> callBlockingMethod(CoprocessorRpcChannel.java:75) > > ~[stormjar.jar:?] > > >> at org.apache.hadoop.hbase.protobuf.generated. > > >> MultiRowMutationProtos$MultiRowMutationService$ > > BlockingStub.mutateRows( > > >> MultiRowMutationProtos.java:2149) ~[stormjar.jar:?] > > >> at ... > > >> ... 13 more > > >> Caused by: org.apache.hadoop.hbase.client. > > RetriesExhaustedException: > > >> Failed after attempts=35, exceptions: > > >> Wed May 23 07:15:57 KST 2018, RpcRetryingCaller{ > > globalStartTime=1527027357808, > > >> pause=100, retries=35}, org.apache.hadoop.hbase. > > RegionTooBusyException: > > >> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore > > limit, > > >> regionName={region}, server={server}, memstoreSize=2600502128, > > >> blockingMemStoreSize=2600468480 > > >> at org.apache.hadoop.hbase.regionserver.HRegion. > > >> checkResources(HRegion.java:3649) > > >> at org.apache.hadoop.hbase.regionserver.HRegion. > > >> processRowsWithLocks(HRegion.java:6935) > > >> at org.apache.hadoop.hbase.regionserver.HRegion. > > >> mutateRowsWithLocks(HRegion.java:6885) > > >> at org.apache.hadoop.hbase.coprocessor. > > MultiRowMutationEndpoint. > > >> mutateRows(MultiRowMutationEndpoint.java:116) > > >> at org.apache.hadoop.hbase.protobuf.generated. > > >> MultiRowMutationProtos$MultiRowMutationService.callMethod( > > >> MultiRowMutationProtos.java:2053) > > >> at org.apache.hadoop.hbase.regionserver.HRegion. > > >> execService(HRegion.java:7875) > > >> at org.apache.hadoop.hbase.regionserver.RSRpcServices. > > >> execServiceOnRegion(RSRpcServices.java:2008) > > >> > > >> > > >> Best regards, > > >> Minwoo Kang > > >> > > >> ________________________________________ > > >> 보낸 사람: 张铎(Duo Zhang) <palomino...@gmail.com> > > >> 보낸 날짜: 2018년 5월 23일 수요일 09:22 > > >> 받는 사람: hbase-user > > >> 제목: Re: can not write to HBase > > >> > > >> What is the exception? And the stacktrace? > > >> > > >> 2018-05-23 8:17 GMT+08:00 Kang Minwoo <minwoo.k...@outlook.com > >: > > >> > > >>> Hello, Users > > >>> > > >>> My HBase client does not work after print the following > logs. > > >>> > > >>> Call exception, tries=23, retries=35, started=291277 ms ago, > > >>> cancelled=false, msg=row '{row}' on table '{table}' at > > region={region}, > > >>> hostname={hostname}, seqNum=100353531 > > >>> > > >>> There are no special logs in the Master and Region Servers. > > >>> Is it something wrong in the client? > > >>> > > >>> Best regards, > > >>> Minwoo Kang > > >>> > > >> > > </minwoo.k...@outlook.com></palomino...@gmail.com></minwoo. > > k...@outlook.com></palomino...@gmail.com></minwoo.k...@outlook.com></ > > anilgupt...@gmail.com></minwoo.k...@outlook.com> >
