[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368648#comment-16368648 ]
Rakesh R commented on HDFS-10285: --------------------------------- We have worked on the comments and following is quick update. {{Comment-1)}} => DONE via HDFS-1309 {{Comment-2)}} => DONE via HDFS-13097 {{Comment-5)}} => DONE via HDFS-13097 {{Comment-8)}} => DONE via HDFS-13097 {{Comment-10)}} => DONE via HDFS-13110 {{Comment-11)}} => DONE via HDFS-13097 {{Comment-12)}} => DONE via HDFS-13097 {{Comment-13)}} => DONE via HDFS-13110 {{Comment-15)}} => DONE via HDFS-13097 {{Comment-16)}} => DONE via HDFS-13097 {{Comment-18)}} => DONE via HDFS-13097 {{Comment-19)}} => DONE via HDFS-13097 {{Comment-22)}} => DONE via HDFS-13097 *For the below comments*, it would be great to hear your thoughts. Please let me know your feedback on my reply. {{Comment-3)}} => This comment has two parts ibr and data transfer. IBR will be explored and implemented via HDFS-13165 sub-task. But, data transfer part is not concluded yet. How do we incorporate local move into this, currently data transfer is not having such logic. IIUC, DNA_TRANSFER is used to send a copy of a block to another datanode. Also, mover tool uses replaceBlock() for block movement, which already has block movement to a different storage within the same datanode. How abt using \{{replaceBlock}} pattern here in sps as well? {{Comment-4}} => Depends on comment-3 {{Comment-6, Comment-9, Comment-14, Comment-17}} => Needs to understand more on this. {{Comment-20}} => Depends on comment-3 *In Progress tasks:* {{Comment-3)}} => HDFS-13165, this jira will only implement logic to collects back the moved block via IBR. {{Comment-21)}} => HDFS-13165 {{Comment-7)}} => HDFS-13166 > Storage Policy Satisfier in Namenode > ------------------------------------ > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode > Affects Versions: HDFS-10285 > Reporter: Uma Maheswara Rao G > Assignee: Uma Maheswara Rao G > Priority: Major > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-10285-consolidated-merge-patch-02.patch, > HDFS-10285-consolidated-merge-patch-03.patch, > HDFS-10285-consolidated-merge-patch-04.patch, > HDFS-10285-consolidated-merge-patch-05.patch, > HDFS-SPS-TestReport-20170708.pdf, SPS Modularization.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org