[DISCUSS] Quadratic Scaling of Max Calls in General Queue with Handler Count

[UMESH KUMAR Kumawat]

Hi!

Recently we encountered an issue where the number of calls in generalQueues
(numCallsInGeneralQueue) was reaching unusually high values. I found that
by default there is a quadratic relation between total number of handlers
and total number of calls in general queues.
Regarding the handler and queues HBase has these config.

   - - hbase.regionserver.handler.count  => Number of handlers, Default is
   30.
   - - hbase.ipc.server.callqueue.handler.factor  => Queue-to-handler
   ratio. Default it is 0.1. Means 10 handlers per queue.
   - - number of call queues = handlerCount * callQueuesHandlersFactor(0.1)
   (code
   
<https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L195>
   )
   - - hbase.ipc.server.max.callqueue.length  => Max callqueue Length.
   Default is DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER(10) * total
   handlerCount  (code
   
<https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java#L71>
   )
      - *   Issue*: I think there’s a small issue here. We are currently
      treating this as the length per queue, but it's calculated using
the *total
      handler count*, not the number of handlers per queue.
   - - In addition we also have a hard limit of (
   DEFAULT_CALL_QUEUE_SIZE_HARD_LIMIT) 250 (code
   
<https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L228>)
   this is hard limit on minimum value of queue. Queue size can not be less
   then 250.

So by default maximum possible number of calls in general Queue =

= number of call queues * hbase.ipc.server.max.callqueue.length
= handlerCount * callQueuesHandlersFactor(0.1) * 10 * handlerCount
= handlerCount^2

 considering DEFAULT_CALL_QUEUE_SIZE_HARD_LIMIT it will be handler *
max(handler * 25)

   1. Is this expected behavior to increase both depth and count of queues ?
   2. I believe we may have unintentionally created a dependency between
   the total number of handlers and the call queue length. Should we
   reconsider this logic?
   3. Regarding the hard limit: should we be enforcing a *minimum* value
   for the queue size or a *maximum* value for the queue size ?


Best Regards,
Umesh