[jira] [Commented] (HDFS-9810) Allow support for more than one block replica per datanode
[ https://issues.apache.org/jira/browse/HDFS-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212300#comment-16212300 ] Ewan Higgs commented on HDFS-9810: -- This should be fixed by HDFS-12685. > Allow support for more than one block replica per datanode > -- > > Key: HDFS-9810 > URL: https://issues.apache.org/jira/browse/HDFS-9810 > Project: Hadoop HDFS > Issue Type: Task > Components: datanode >Reporter: Virajith Jalaparti > > Datanodes report and store only one replica of each block. It should be > possible to store multiple replicas among its different configured storage > types, particularly to support non-durable media and remote storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9810) Allow support for more than one block replica per datanode
[ https://issues.apache.org/jira/browse/HDFS-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149565#comment-15149565 ] Zhe Zhang commented on HDFS-9810: - Thanks clarifying this Chris. Makes sense to me. > Allow support for more than one block replica per datanode > -- > > Key: HDFS-9810 > URL: https://issues.apache.org/jira/browse/HDFS-9810 > Project: Hadoop HDFS > Issue Type: Task > Components: datanode >Reporter: Virajith Jalaparti > > Datanodes report and store only one replica of each block. It should be > possible to store multiple replicas among its different configured storage > types, particularly to support non-durable media and remote storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9810) Allow support for more than one block replica per datanode
[ https://issues.apache.org/jira/browse/HDFS-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149475#comment-15149475 ] Chris Douglas commented on HDFS-9810: - Agreed, HDFS-9806 will have a feature branch. This was raised a a separate issue rather than a subtask to share a solution with non-durable replicas (HDFS-5851, HDFS-6122), which have many of the same properties. It will also track the work on intra-DN rebalancing, which should also be resilient to multiple replicas of the same block (temporarily, at least during restart/upgrade). If the requirements are unique to HDFS-9806 then we can change the scope, but to the extent that this case already exists we wanted to discuss how to handle it correctly. For example, if intra-DN rebalancing does not admit the possibility of multiple replicas, then HDFS-9806 could implement the same logic (i.e., replace the remote replica with a local one). The intent of this JIRA is to be consistent, not prescribe a particular solution. > Allow support for more than one block replica per datanode > -- > > Key: HDFS-9810 > URL: https://issues.apache.org/jira/browse/HDFS-9810 > Project: Hadoop HDFS > Issue Type: Task > Components: datanode >Reporter: Virajith Jalaparti > > Datanodes report and store only one replica of each block. It should be > possible to store multiple replicas among its different configured storage > types, particularly to support non-durable media and remote storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9810) Allow support for more than one block replica per datanode
[ https://issues.apache.org/jira/browse/HDFS-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149232#comment-15149232 ] Zhe Zhang commented on HDFS-9810: - [~chris.douglas] / [~virajith] I had a similar confusion when seeing this JIRA come up. Then I realized it's part of the HDFS-9806 work. Given the scope of HDFS-9806 it will most likely need a feature branch. To couple with that, maybe we should make HDFS-9806 an umbrella JIRA and convert this series of JIRAs as subtasks? > Allow support for more than one block replica per datanode > -- > > Key: HDFS-9810 > URL: https://issues.apache.org/jira/browse/HDFS-9810 > Project: Hadoop HDFS > Issue Type: Task > Components: datanode >Reporter: Virajith Jalaparti > > Datanodes report and store only one replica of each block. It should be > possible to store multiple replicas among its different configured storage > types, particularly to support non-durable media and remote storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9810) Allow support for more than one block replica per datanode
[ https://issues.apache.org/jira/browse/HDFS-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146194#comment-15146194 ] Virajith Jalaparti commented on HDFS-9810: -- Our use case is driven by HDFS-9806 -- the idea is when a "provided" block is fetched by a datanode, the datanode can cache it on local storage media (disk, SSD etc.) so that future reads of the data are served from it. When the block is cached, the datanode has to maintain information for two replicas (in FSDatasetImpl), one for the local replica and the other for the "provided" replica. With this, if the local replica is lost, it can still be accessed from the provided replica. This is a modification in the FSDatasetImpl to enable the possibility of a datanode storing multiple replicas for a block. We are not proposing to change the BlockPlacementPolicyDefault, which determines how the replicas are placed. The BlockPlacementPolicyDefault will still work as it works today and hence, the failure model will remain the same. > Allow support for more than one block replica per datanode > -- > > Key: HDFS-9810 > URL: https://issues.apache.org/jira/browse/HDFS-9810 > Project: Hadoop HDFS > Issue Type: Task > Components: datanode >Reporter: Virajith Jalaparti > > Datanodes report and store only one replica of each block. It should be > possible to store multiple replicas among its different configured storage > types, particularly to support non-durable media and remote storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
