No, there are no other quotas set. The list_quota shows only REQUEST_TYPE as
set, which is what I want.
Not sure why it tries validating READ_SIZE when I have set only REQUEST_TYPE.
Is anyone aware of known issues with Hbase 1.1? Setting separate read and write
quotas was only introduced in 1.2, but not sure why this quota exceeded error
with READ_SIZE then shows up.
Thanks,Sumit
From: Guanghao Zhang <[email protected]>
To: [email protected]; Sumit Nigam <[email protected]>
Sent: Thursday, September 22, 2016 12:49 PM
Subject: Re: Hbase throttling issues
Did you set any other quotas? The exception
ThrottlingException.throwReadSizeExceeded
shows that the throttle type is READ_SIZE, but your quota's throttle type
is REQUEST_SIZE. You can use list_quota shell cmd to check whether there
are READ_SIZE quota.
2016-09-22 12:06 GMT+08:00 Sumit Nigam <[email protected]>:
> No, did not try request number. I want to use size as my throttling
> factor.
>
> Thanks again!
>
> From: Guanghao Zhang <[email protected]>
> To: [email protected]; Sumit Nigam <[email protected]>
> Sent: Thursday, September 22, 2016 9:28 AM
> Subject: Re: Hbase throttling issues
>
> Did you try to use REQUEST_NUMBER as throttle type?
>
> 2016-09-22 11:33 GMT+08:00 Sumit Nigam <[email protected]>:
>
> > Hi Guanghao,
> >
> > This throttling error shows up as soon as I start hbase. So, ideally
> there
> > shouldn't be too many prior operations in play here. Plus, the error
> shows
> > up even when my table has hardly any data (possibly in kbs) and I have
> set
> > throttling limit to be ~10GB.
> >
> > Thanks,Sumit
> >
> > From: Guanghao Zhang <[email protected]>
> > To: [email protected]; Sumit Nigam <[email protected]>
> > Sent: Thursday, September 22, 2016 7:37 AM
> > Subject: Re: Hbase throttling issues
> >
> > All scan operations were throttled? Now it use the avg size of all
> previous
> > operations to check quota. Maybe the previous scan operation read too
> much
> > data.
> >
> > 2016-09-22 1:18 GMT+08:00 Sumit Nigam <[email protected]>:
> >
> > > Hi Hbase experts,
> > >
> > > Our application is unable to scan or read from hbase tables, when
> > > throttling is set. We are getting ThrottlingException every time. The
> > error
> > > is seen more frequently when the number of hbase pre splits is
> > > increased. The size tables for which this error is showing is empty (
> > > during some runs it was very low in the order of few kbs). Have tried
> > both
> > > rate limiters already - average and fixed. Can't understand why read
> rate
> > > limit exceeds when there is hardly any data in hbase. Anyone has faced
> > this
> > > issue before?
> > >
> > > Setup Details:
> > >
> > > Hbase version : 1.1.2
> > > Number of region servers :4
> > > Number of regions : 116
> > > HeapMemory for Region Server : 2GB
> > >
> > > Quotas set :
> > > TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE,
> > LIMIT
> > > => 10G/sec, SCOPE => MACHINE
> > > TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE,
> > LIMIT
> > > => 10G/sec, SCOPE => MACHINE
> > >
> > > Following is the error we faced. Pasting debug logs of regions server:
> > >
> > > 2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler=
> > 55,queue=1,port=58526]
> > > quotas.RegionServerQuotaManager: Throttling exception for user=root
> > > table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit
> > > exceeded - wait 0.00sec
> > >
> > > 2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler=
> > 55,queue=1,port=58526]
> > > ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526:
> callId:
> > > 52 service: ClientService methodName: Scan size: 28 connection:
> > > 10.65.141.170:42806
> > >
> > > org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit
> > > exceeded - wait 0.00sec
> > >
> > > at org.apache.hadoop.hbase.quotas.ThrottlingException.
> > > throwThrottlingException(ThrottlingException.java:107)
> > >
> > > at org.apache.hadoop.hbase.quotas.ThrottlingException.
> > > throwReadSizeExceeded(ThrottlingException.java:101)
> > >
> > > at org.apache.hadoop.hbase.quotas.TimeBasedLimiter.
> > > checkQuota(TimeBasedLimiter.java:139)
> > >
> > > at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota(
> > > DefaultOperationQuota.java:59)
> > >
> > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(
> > > RegionServerQuotaManager.java:180)
> > >
> > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(
> > > RegionServerQuotaManager.java:125)
> > >
> > > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > > scan(RSRpcServices.java:2265)
> > >
> > > at org.apache.hadoop.hbase.protobuf.generated.
> > > ClientProtos$ClientService$2.callBlockingMethod(
> ClientProtos.java:32205)
> > >
> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> > >
> > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> > >
> > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> > > RpcExecutor.java:130)
> > >
> > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > > Thanks,
> > >
> > > Sumit
> >
> >
> >
> >
>
>
>
>
