[
https://issues.apache.org/jira/browse/HDFS-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037827#comment-17037827
]
Jinglun edited comment on HDFS-15087 at 2/16/20 1:18 PM:
---------------------------------------------------------
Hi [~brahmareddy] [~hexiaoqiao], thanks your comments. Very sorry for my late
response! I'm working on the initial pull request now.
Reply to Xiaoqiao:
{quote}For step saveTree, it generates 2 files, TREE-FILE and TREE-META
respectively, I don't get the difference with one meta file with all
information. My concerns is we should skip to check consistency and other
sanity checks if keep only one meta file.
{quote}
The TREE-META file is used for sanity check so we don't need to scan the
TREE-FILE to collect the meta information.
{quote}It may be not enough to ensure source directory not changes if only
revoke write permission for source directory/file since some super user action
actually not do permission check when do read/write ops. +1 for adding extra
xattributes to refuse write operation request.
{quote}
Agree with you.
{quote}Any consideration about HA mode?
{quote}
The files are written in a shared storage like an HDFS. And the consistency
should be ok as we have the 2-phase editlogs.
{quote}Some user cases do not shared DN's in federated clusters as Ayush Saxena
mentioned above, it needs to rollback to block transfer if use hard-link
solution?
{quote}
Yes. We have the HFR(distcp) to handle the block transfer.
{quote}It introduces Scheduler and Externel Storage modules in design doc, what
about use Router to schedule rename task and NN local storage (may be it is
same as fsimage persist path) to keep meta data then we could not need extra
module and reduce maintain cost.
{quote}
This is the future plan: the Scheduler should be running in the Router. To keep
simple at the initial patch I'll only support Scheduler running as a shell
command.
[请尝试网页搜索|http://www.youdao.com/search?q=the%20initial%20patch&ue=utf8&keyfrom=chrome.extension]
was (Author: lijinglun):
Hi [~brahmareddy] [~hexiaoqiao], thanks your comments. Very sorry for my late
response! I'm working on the initial pull request now.
Reply to Xiaoqiao:
{quote}For step saveTree, it generates 2 files, TREE-FILE and TREE-META
respectively, I don't get the difference with one meta file with all
information. My concerns is we should skip to check consistency and other
sanity checks if keep only one meta file.
{quote}
The TREE-META file is used for sanity check so we don't need to scan the
TREE-FILE to collect the meta information.
{quote}
It may be not enough to ensure source directory not changes if only revoke
write permission for source directory/file since some super user action
actually not do permission check when do read/write ops. +1 for adding extra
xattributes to refuse write operation request.
{quote}
Agree with you.
{quote}Any consideration about HA mode?
{quote}
The files are written in a shared storage like an HDFS. And the consistency
should be ok as we have the 2-phase editlogs.
{quote}Some user cases do not shared DN's in federated clusters as Ayush Saxena
mentioned above, it needs to rollback to block transfer if use hard-link
solution?
{quote}
Yes. We have the HFR(distcp) to handle the block transfer.
{quote}It introduces Scheduler and Externel Storage modules in design doc, what
about use Router to schedule rename task and NN local storage (may be it is
same as fsimage persist path) to keep meta data then we could not need extra
module and reduce maintain cost.
{quote}
This is the future plan: the Scheduler should be running in the Router. To keep
simple at the initial patch I'll only support Scheduler running as a shell
command.
[the initial
patch|http://dict.youdao.com/search?q=the%20initial%20patch&keyfrom=chrome.extension]
[详细|http://www.youdao.com/search?q=the%20initial%20patch&ue=utf8&keyfrom=chrome.extension]X
没有英汉互译结果
[请尝试网页搜索|http://www.youdao.com/search?q=the%20initial%20patch&ue=utf8&keyfrom=chrome.extension]
> RBF: Balance/Rename across federation namespaces
> ------------------------------------------------
>
> Key: HDFS-15087
> URL: https://issues.apache.org/jira/browse/HDFS-15087
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jinglun
> Priority: Major
> Attachments: HDFS-15087.initial.patch, HFR_Rename Across Federation
> Namespaces.pdf
>
>
> The Xiaomi storage team has developed a new feature called HFR(HDFS
> Federation Rename) that enables us to do balance/rename across federation
> namespaces. The idea is to first move the meta to the dst NameNode and then
> link all the replicas. It has been working in our largest production cluster
> for 2 months. We use it to balance the namespaces. It turns out HFR is fast
> and flexible. The detail could be found in the design doc.
> Looking forward to a lively discussion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
