[ https://issues.apache.org/jira/browse/HDFS-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037827#comment-17037827 ]
Jinglun commented on HDFS-15087: -------------------------------- Hi [~brahmareddy] [~hexiaoqiao], thanks your comments. Very sorry for my late response! I'm working on the initial pull request now. Reply to Xiaoqiao: {quote}For step saveTree, it generates 2 files, TREE-FILE and TREE-META respectively, I don't get the difference with one meta file with all information. My concerns is we should skip to check consistency and other sanity checks if keep only one meta file. {quote} The TREE-META file is used for sanity check so we don't need to scan the TREE-FILE to collect the meta information. {quote} It may be not enough to ensure source directory not changes if only revoke write permission for source directory/file since some super user action actually not do permission check when do read/write ops. +1 for adding extra xattributes to refuse write operation request. {quote} Agree with you. {quote}Any consideration about HA mode? {quote} The files are written in a shared storage like an HDFS. And the consistency should be ok as we have the 2-phase editlogs. {quote}Some user cases do not shared DN's in federated clusters as Ayush Saxena mentioned above, it needs to rollback to block transfer if use hard-link solution? {quote} Yes. We have the HFR(distcp) to handle the block transfer. {quote}It introduces Scheduler and Externel Storage modules in design doc, what about use Router to schedule rename task and NN local storage (may be it is same as fsimage persist path) to keep meta data then we could not need extra module and reduce maintain cost. {quote} This is the future plan: the Scheduler should be running in the Router. To keep simple at the initial patch I'll only support Scheduler running as a shell command. [the initial patch|http://dict.youdao.com/search?q=the%20initial%20patch&keyfrom=chrome.extension] [详细|http://www.youdao.com/search?q=the%20initial%20patch&ue=utf8&keyfrom=chrome.extension]X 没有英汉互译结果 [请尝试网页搜索|http://www.youdao.com/search?q=the%20initial%20patch&ue=utf8&keyfrom=chrome.extension] > RBF: Balance/Rename across federation namespaces > ------------------------------------------------ > > Key: HDFS-15087 > URL: https://issues.apache.org/jira/browse/HDFS-15087 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Jinglun > Priority: Major > Attachments: HDFS-15087.initial.patch, HFR_Rename Across Federation > Namespaces.pdf > > > The Xiaomi storage team has developed a new feature called HFR(HDFS > Federation Rename) that enables us to do balance/rename across federation > namespaces. The idea is to first move the meta to the dst NameNode and then > link all the replicas. It has been working in our largest production cluster > for 2 months. We use it to balance the namespaces. It turns out HFR is fast > and flexible. The detail could be found in the design doc. > Looking forward to a lively discussion. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org