[
https://issues.apache.org/jira/browse/HDFS-11023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607010#comment-15607010
]
Andrew Wang commented on HDFS-11023:
------------------------------------
I looked at our existing replication controls in more detail, and EC is being
correctly accounted for. The limits are on outgoing streams from the source
nodes, and EC recovery work has multiple source DNs. So nothing to do
immediately.
I think the potential improvements here would be toward making the replication
controls easier to configure from an admin POV. Some ideas:
* Have NN throttles based on the # of bytes.
* Do byte-based throttling on the DN side. This will be more accurate and
easier for admins to understand.
* Have a DN-side # of streams limit. I think this could replace the NN-side
#-streams-per-DN limit, since the DN has better awareness of the # of open TCP
connections than the NN.
* Dynamically size the NN-side throttles per DN. This would be like TCP window
sizing, and would let us more reliably avoid congested DNs. Right now it's done
with retries and randomness.
> I/O based throttling of DN replication work
> -------------------------------------------
>
> Key: HDFS-11023
> URL: https://issues.apache.org/jira/browse/HDFS-11023
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.0.0-alpha1
> Reporter: Andrew Wang
>
> Datanode recovery work is currently throttled based on the number of blocks
> to be recovered. This is fine in a replicated world, since each block is
> equally expensive to recover. However, EC blocks are much more expensive to
> recover, and the amount depends on the EC policy.
> It'd be better to have recovery throttles that accounted for the amount of
> I/O rather than just the # of blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
