[ https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Priyank Rastogi updated HDFS-8041: ---------------------------------- Labels: (was: BB2015-05-TBR) > Consider remaining space during block blockplacement if dfs space is highly > utilized > ------------------------------------------------------------------------------------ > > Key: HDFS-8041 > URL: https://issues.apache.org/jira/browse/HDFS-8041 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Kihwal Lee > Assignee: Kihwal Lee > Attachments: HDFS-8041.v1.patch, HDFS-8041.v2.patch, > HDFS-8041.v3.patch, HDFS-8041.v4.patch > > > This feature is helpful in avoiding smaller nodes (i.e. heterogeneous > environment) getting constantly being full when the overall space utilization > is over a certain threshold. When the utilization is low, balancer can keep > up, but once the average per-node byte goes over the capacity of the smaller > nodes, they get full so quickly even after perfect balance. > This jira proposes an improvement that can be optionally enabled in order to > slow down the rate of space usage growth of smaller nodes if the overall > storage utilization is over a configured threshold. It will not replace > balancer, rather will help balancer keep up. Also, the primary replica > placement will not be affected. Only the replicas typically placed in a > remote rack will be subject to this check. > The appropriate threshold is cluster configuration specific. There is no > generally good value to set, thus it is disabled by default. We have seen > cases where the threshold of 85% - 90% would help. Figuring when > {{totalSpaceUsed / numNodes}} becomes close to the capacity of a smaller node > is helpful in determining the threshold. -- This message was sent by Atlassian JIRA (v6.3.4#6332)