[ https://issues.apache.org/jira/browse/HDFS-9014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Max Lapan resolved HDFS-9014. ----------------------------- Resolution: Duplicate Release Note: Looks like a duplicate of HDFS-8131 and HDFS-8041 > Block placement policy with respect to DN free space > ---------------------------------------------------- > > Key: HDFS-9014 > URL: https://issues.apache.org/jira/browse/HDFS-9014 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode > Reporter: Max Lapan > Attachments: ReplicationPolicyFreeSpace.patch > > > Default block allocation policy (also known as 'replication policy') > implemented in NN is random selection from suitable candidates (rack-local or > 'other rack'). This is ok when all DNs in a cluster has nearly equal amount > of storage, but leads to problems when some DNs are significantly larger than > other. In that situation, when NN places new blocks in random fashion, extra > space becomes almost unusable and, in extreme case can lead to 100% usage of > all other 'small' DNs with almost empty 'large', which can lead to various > HDFS and MR problems. > Situation when we have datanodes of different sizes is quite real in large, > long-lived systems when different generations of machines are put in a single > cluster. > To overcome this, I implemented a different block allocation policy which > places blocks with respect to free space available on a DN. Please, consider > it for inclusion in hdfs codebase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)