[
https://issues.apache.org/jira/browse/HDFS-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177543#comment-14177543
]
Haohui Mai commented on HDFS-7270:
----------------------------------
The point of this jira is to make the pipeline more stable and reduce
unnecessary aborts / recovery. An alternative approach is to implement
admission control -- HDFS-7265 proposes to introduce a throttler to limit the
amount of the data that is written into HDFS.
Deriving the right configuration for the throttler to balance between the
stability and throughput of the pipeline, however, is difficult in practice.
The loads of the clusters varies from time to time, and the DNs can go ups and
downs which can make the configuration suboptimal thus defeat its purpose.
> Implementing congestion control in writing pipeline
> ---------------------------------------------------
>
> Key: HDFS-7270
> URL: https://issues.apache.org/jira/browse/HDFS-7270
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Haohui Mai
> Assignee: Haohui Mai
>
> When a client writes to HDFS faster than the disk bandwidth of the DNs, it
> saturates the disk bandwidth and put the DNs unresponsive. The client only
> backs off by aborting / recovering the pipeline, which leads to failed writes
> and unnecessary pipeline recovery.
> This jira proposes to add explicit congestion control mechanisms in the
> writing pipeline.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
