Junping Du created HDFS-5183:
--------------------------------
Summary: Combine ReplicaPlacementPolicy with VolumeChoosingPolicy
together to have a global view in choosing DN storage for replica.
Key: HDFS-5183
URL: https://issues.apache.org/jira/browse/HDFS-5183
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode, performance
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Junping Du
Per discussion in HDFS-5157, There are two different ways to handle
BlockPlacementPolicy and ReplicaChoosingPolicy in case of multiple storage
types:
1. Client specifies the required storage type when calling addBlock(..) to NN.
BlockPlacementPolicy in NN chooses a set of datanodes accounting for the
storage type. Then, client passes the required storage type to the datanode set
and each datanode chooses a particular storage using a VolumeChoosingPolicy.
2. Same as before, client specifies the required storage type when calling
addBlock(..) to NN. Now, BlockPlacementPolicy in NN chooses a set of storages
(instead of datanodes). Then, client writes to the corresponding storages.
VolumeChoosingPolicy is no longer needed and it should be removed.
We think #2 is more powerful as it will bring global view to volume choosing or
bring storage status into consideration in replica choosing, so we propose to
combine two polices together.
One concern here is it may increase the load of NameNode as previously volume
choosing is decided by DN. We may verify it later (that's why I put performance
in component).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
