[
https://issues.apache.org/jira/browse/HDFS-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13764311#comment-13764311
]
Eric Sirianni commented on HDFS-5183:
-
Copying over my feedback from HDFS-5157:
bq. I would favor a model like #2 but allow for the DataNode to override the
placement decision (where the override was likely only done in exceptional
circumstances).
This seems consistent with the overarching design of HDFS to allow for loose
synchronization in replica maps between the NameNode and DataNode
(re-synchronized periodically by block reports).
In particular, if the DataNode cannot honor the specific StorageID chosen by
the NameNode (but can still honor the write), I believe this should *not* be an
error condition. Do you all agree?
Combine ReplicaPlacementPolicy with VolumeChoosingPolicy together to have a
global view in choosing DN storage for replica.
---
Key: HDFS-5183
URL: https://issues.apache.org/jira/browse/HDFS-5183
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode, performance
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Junping Du
Per discussion in HDFS-5157, There are two different ways to handle
BlockPlacementPolicy and ReplicaChoosingPolicy in case of multiple storage
types:
1. Client specifies the required storage type when calling addBlock(..) to
NN. BlockPlacementPolicy in NN chooses a set of datanodes accounting for the
storage type. Then, client passes the required storage type to the datanode
set and each datanode chooses a particular storage using a
VolumeChoosingPolicy.
2. Same as before, client specifies the required storage type when calling
addBlock(..) to NN. Now, BlockPlacementPolicy in NN chooses a set of storages
(instead of datanodes). Then, client writes to the corresponding storages.
VolumeChoosingPolicy is no longer needed and it should be removed.
We think #2 is more powerful as it will bring global view to volume choosing
or bring storage status into consideration in replica choosing, so we propose
to combine two polices together.
One concern here is it may increase the load of NameNode as previously volume
choosing is decided by DN. We may verify it later (that's why I put
performance in component).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
