[
https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10967:
-----------------------------
Attachment: HDFS-10967.02.patch
Thanks [~mingma] for the feedback!
Attaching v2 patch to add dfsAdmin command to update the config without NN
restart.
bq. For balancer or over replicated scenarios, chooseReplicasToDelete uses
absolute free space, maybe that needs to be change to percentage based?
Agreed. In general I think all capacity-based decisions (Balancer, placement
policy etc) should be consistent.
bq. What if we move this new policy to isGoodDatanode?
That's a good thought. The challenge is that this capacity consideration is not
a _hard_ restriction. I.e. if a DN's capacity usage if over the factor, it is
not an ideal candidate from balancing perspective, but if there's no other
valid candidates, it should still be considered. I think I found a way to apply
the logic to remote writer and {{BlockPlacementPolicyRackFaultTolerant}}. But
it requires some additional refactors. Will attach v3 patch soon.
> Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
> -----------------------------------------------------------------------
>
> Key: HDFS-10967
> URL: https://issues.apache.org/jira/browse/HDFS-10967
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Zhe Zhang
> Assignee: Zhe Zhang
> Labels: balancer
> Attachments: HDFS-10967.00.patch, HDFS-10967.01.patch,
> HDFS-10967.02.patch
>
>
> Large production clusters are likely to have heterogeneous nodes in terms of
> storage capacity, memory, and CPU cores. It is not always possible to
> proportionally ingest data into DataNodes based on their remaining storage
> capacity. Therefore it's possible for a subset of DataNodes to be much closer
> to full capacity than the rest.
> This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks of
> low-storage nodes and _n_ whole racks of high-storage nodes. So It'd be very
> useful if we can lower the chance for those near-full DataNodes to become
> destinations for the 2nd and 3rd replicas.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
