All scan operations were throttled? Now it use the avg size of all previous operations to check quota. Maybe the previous scan operation read too much data.
2016-09-22 1:18 GMT+08:00 Sumit Nigam <[email protected]>: > Hi Hbase experts, > > Our application is unable to scan or read from hbase tables, when > throttling is set. We are getting ThrottlingException every time. The error > is seen more frequently when the number of hbase pre splits is > increased. The size tables for which this error is showing is empty ( > during some runs it was very low in the order of few kbs). Have tried both > rate limiters already - average and fixed. Can't understand why read rate > limit exceeds when there is hardly any data in hbase. Anyone has faced this > issue before? > > Setup Details: > > Hbase version : 1.1.2 > Number of region servers :4 > Number of regions : 116 > HeapMemory for Region Server : 2GB > > Quotas set : > TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT > => 10G/sec, SCOPE => MACHINE > TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT > => 10G/sec, SCOPE => MACHINE > > Following is the error we faced. Pasting debug logs of regions server: > > 2016-09-17 22:35:40,674 DEBUG > [B.defaultRpcServer.handler=55,queue=1,port=58526] > quotas.RegionServerQuotaManager: Throttling exception for user=root > table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit > exceeded - wait 0.00sec > > 2016-09-17 22:35:40,676 DEBUG > [B.defaultRpcServer.handler=55,queue=1,port=58526] > ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId: > 52 service: ClientService methodName: Scan size: 28 connection: > 10.65.141.170:42806 > > org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit > exceeded - wait 0.00sec > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > throwThrottlingException(ThrottlingException.java:107) > > at org.apache.hadoop.hbase.quotas.ThrottlingException. > throwReadSizeExceeded(ThrottlingException.java:101) > > at org.apache.hadoop.hbase.quotas.TimeBasedLimiter. > checkQuota(TimeBasedLimiter.java:139) > > at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota( > DefaultOperationQuota.java:59) > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota( > RegionServerQuotaManager.java:180) > > at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota( > RegionServerQuotaManager.java:125) > > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > scan(RSRpcServices.java:2265) > > at org.apache.hadoop.hbase.protobuf.generated. > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( > RpcExecutor.java:130) > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > > at java.lang.Thread.run(Thread.java:745) > > > Thanks, > > Sumit
