[
https://issues.apache.org/jira/browse/HDFS-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588686#comment-14588686
]
Colin Patrick McCabe commented on HDFS-7923:
--------------------------------------------
BTW it would be relatively easy to enforce "fairness" on the NN side in this
patch. In fact I had an earlier version that did do that. It would give out
FBR leases to datanodes based on a round-robin order. But I decided against it
since it was extra complexity without a good reason. In particular, I was
worried that I'd create starvation if the set of DNs the NN selected to report
in didn't want to report in (perhaps because they were using initialDelay...).
Since full block report periods can be very long this is a real concern and
would necessitate a hack like a timeout. It was much easier to keep block
report scheduling as a datanode-side thing.
We can revisit this if it ever seems like a good idea, but I bet that you will
not be able to create starvation in any real cluster without creating
conditions that would make the cluster unusable no matter what.
> The DataNodes should rate-limit their full block reports by asking the NN on
> heartbeat messages
> -----------------------------------------------------------------------------------------------
>
> Key: HDFS-7923
> URL: https://issues.apache.org/jira/browse/HDFS-7923
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: 2.8.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Fix For: 2.8.0
>
> Attachments: HDFS-7923.000.patch, HDFS-7923.001.patch,
> HDFS-7923.002.patch, HDFS-7923.003.patch, HDFS-7923.004.patch,
> HDFS-7923.006.patch, HDFS-7923.007.patch
>
>
> The DataNodes should rate-limit their full block reports. They can do this
> by first sending a heartbeat message to the NN with an optional boolean set
> which requests permission to send a full block report. If the NN responds
> with another optional boolean set, the DN will send an FBR... if not, it will
> wait until later. This can be done compatibly with optional fields.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
