[ https://issues.apache.org/jira/browse/HDFS-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123173#comment-14123173 ]
Chris Nauroth commented on HDFS-6967: ------------------------------------- This is another case where it's a real shame that we're stuck on Jetty 6. Starting in Jetty 7, it's possible to limit the queue that backs the {{QueuedThreadPool}}. Here is the relevant constructor in current Jetty 9: http://download.eclipse.org/jetty/stable-9/apidocs/org/eclipse/jetty/util/thread/QueuedThreadPool.html#QueuedThreadPool(int, int, int, java.util.concurrent.BlockingQueue) [~daryn], I know this has always been a touchy subject, but do you think this problem warrants reopening discussion of a Jetty upgrade? I believe that would set us up for a relatively simple fix in {{HttpServer2#initializeWebServer}}, perhaps with some new config properties to control queue size. > DNs may OOM under high webhdfs load > ----------------------------------- > > Key: HDFS-6967 > URL: https://issues.apache.org/jira/browse/HDFS-6967 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, webhdfs > Affects Versions: 2.0.0-alpha, 3.0.0 > Reporter: Daryn Sharp > Assignee: Eric Payne > > Webhdfs uses jetty. The size of the request thread pool is limited, but > jetty will accept and queue infinite connections. Every queued connection is > "heavy" with buffers, etc. Unlike data streamer connections, thousands of > webhdfs connections will quickly OOM a DN. The accepted requests must be > bounded and excess clients rejected so they retry on a new DN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)