[
https://issues.apache.org/jira/browse/HDFS-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180784#comment-14180784
]
Haohui Mai commented on HDFS-7279:
----------------------------------
An alternative option is to upgrade jetty and servlet. The New APIs from both
jetty and servlet such as asynchronous servlet can amend some of the issues.
Webhdfs on the DN side, however, is data intensive which does not fit the
servlet API very well. The servlet / jetty APIs do not give fine-grain control
on the resources that netty is able to provide. These controls are critical if
webhdfs needs to survive on heavy workload.
The strategy is proven by the mapreduce client, which already uses netty to
implement the shuffle functionality. For other URLs on the DNs, I plan to keep
jetty listening on a local address, but to have a reverse proxy in netty to
continue the serve these URLs.
> Use netty to implement DatanodeWebHdfsMethods
> ---------------------------------------------
>
> Key: HDFS-7279
> URL: https://issues.apache.org/jira/browse/HDFS-7279
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Haohui Mai
> Assignee: Haohui Mai
>
> Currently the DN implements all related webhdfs functionality using jetty. As
> the current jetty version the DN used (jetty 6) lacks of fine-grained buffer
> and connection management, DN often suffers from long latency and OOM when
> its webhdfs component is under sustained heavy load.
> This jira proposes to implement the webhdfs component in DN using netty,
> which can be more efficient and allow more finer-grain controls on webhdfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
