[ https://issues.apache.org/jira/browse/HBASE-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17560816#comment-17560816 ]
Michael Stack commented on HBASE-27112: --------------------------------------- {quote}Instrument netty level resources to understand better actual resource allocations under load. Investigate what we need to plug in where to gain visibility. {quote} There doesn't seem to be an amenable, native metrics export by netty and core classes/interfaces are bare of anything but functionality so tough getting counts w/o gymnastics (It seems like it is an old ask of netty [https://github.com/netty/netty/issues/6523]. or https://groups.google.com/g/netty/c/-ZNx-L75csc/m/z8u5rp6lCUQJ). Thread-dumping while under load w/ default config., I see 2*CPU_COUNT netty RS-EventLoopGroup-N threads all RUNNABLE seemingly doing nothing stuck on epoll.Native.epollWait. I am running w/ 2*CPU_COUNT handlers. Netty thread count does seems excessive. I tried running w/ 8 threads in the RS-EventLoopGroup pool on a 24 "CPU" node w/ load on and throughput seemed less, but pretty close; 75th percentile seemed the same before as after but 99.9th percentile was elevated some (~70ms vs ~90ms). With less threads I started to get a few CallQueueTooBigExceptions on a few servers which I didn't notice happening when 2*CPU_COUNT. I tried with CPU_COUNT netty RS-EventLoopGroup threads. All seems to be about same as 2*CPU_COUNT – perhaps slightly less throughput though the 99.9th percentile seemed less (~70ms vs ~50). There were a few transient CallQueueTooBigExceptions. I think I'm going to leave the default 2*CPU_COUNT in place for now on this cluster though it seems profligate (80% read/20% write) until someone does a deeper dig than the cursory one done here. Thanks. Hope this helps. > Investigate Netty resource usage limits > --------------------------------------- > > Key: HBASE-27112 > URL: https://issues.apache.org/jira/browse/HBASE-27112 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC > Affects Versions: 2.5.0 > Reporter: Andrew Kyle Purtell > Assignee: Andrew Kyle Purtell > Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-4 > > > We leave Netty level resource limits unbounded. The number of threads to use > for the event loop is default 0 (unbounded). The default for > io.netty.eventLoop.maxPendingTasks is INT_MAX. > We don't do that for our own RPC handlers. We have a notion of maximum > handler pool size, with a default of 30, typically raised in production by > the user. We constrain the depth of the request queue in multiple ways... > limits on number of queued calls, limits on total size of calls data that can > be queued (to avoid memory usage overrun, CoDel conditioning of the call > queues if it is enabled, and so on. > Under load can we pile up a excess of pending request state, such as direct > buffers containing request bytes, at the netty layer because of downstream > resource limits? Those limits will act as a bottleneck, as intended, and > before would have also applied backpressure through RPC too, because > SimpleRpcServer had thread limits ("hbase.ipc.server.read.threadpool.size", > default 10), but Netty may be able to queue up a lot more, in comparison, > because Netty has been optimized to prefer concurrency. > Consider the hbase.netty.eventloop.rpcserver.thread.count default. It is 0 > (unbounded). I don't know what it can actually get up to in production, > because we lack the metric, but there are diminishing returns when threads > > cores so a reasonable default here could be > Runtime.getRuntime().availableProcessors() instead of unbounded? > maxPendingTasks probably should not be INT_MAX, but that may matter less. > The tasks here are: > - Instrument netty level resources to understand better actual resource > allocations under load. Investigate what we need to plug in where to gain > visibility. > - Where instrumentation designed for this issue can be implemented as low > overhead metrics, consider formally adding them as a metric. > - Based on the findings from this instrumentation, consider and implement > next steps. The goal would be to limit concurrency at the Netty layer in such > a way that performance is still good, and under load we don't balloon > resource usage at the Netty layer. > If the instrumentation and experimental results indicate no changes are > necessary, we can close this as Not A Problem or WontFix. -- This message was sent by Atlassian Jira (v8.20.10#820010)