[
https://issues.apache.org/jira/browse/HDFS-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239955#comment-15239955
]
Jing Zhao commented on HDFS-9820:
---------------------------------
Thanks again for working on this, Yongjun. Some comments:
# Even without HDFS-10263, looks like we can still achieve the goal by
computing the diff with reversed snapshot names. I guess we only need to skip
{{DistCpSync#prepareDiffList}} here. Can you confirm? Or do you see any other
issue?
# When using snapshot diff for distcp, our basic assumption is "there is no
change between snapshot {{from}} and the current status in target
file system." This is our best effort to avoid applying the diff on a wrong
state of the target. Now with the patch this check is disabled for rdiff.
However, in the rdiff scenario, the current status of target is always mapped
to a snapshot in both source and target, thus looks to me we do not need to
make any changes in command options.
# Because of HDFS-10263, we only need to understand which snapshot is earlier,
the {{from}} or {{to}}. This information can be retrieved from the modification
time of the two snapshots (i.e., to call {{getListing} against
{{path-of-snapshottable-dir/.snapshot}}}).
> Improve distcp to support efficient restore to an earlier snapshot
> ------------------------------------------------------------------
>
> Key: HDFS-9820
> URL: https://issues.apache.org/jira/browse/HDFS-9820
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: distcp
> Reporter: Yongjun Zhang
> Assignee: Yongjun Zhang
> Attachments: HDFS-9820.001.patch, HDFS-9820.002.patch,
> HDFS-9820.003.patch, HDFS-9820.004.patch
>
>
> HDFS-4167 intends to restore HDFS to the most recent snapshot, and there are
> some complexity and challenges.
> HDFS-7535 improved distcp performance by avoiding copying files that changed
> name since last backup.
> On top of HDFS-7535, HDFS-8828 improved distcp performance when copying data
> from source to target cluster, by only copying changed files since last
> backup. The way it works is use snapshot diff to find out all files changed,
> and copy the changed files only.
> See
> https://blog.cloudera.com/blog/2015/12/distcp-performance-improvements-in-apache-hadoop/
> This jira is to propose a variation of HDFS-8828, to find out the files
> changed in target cluster since last snapshot sx, and copy these from the
> source target's same snapshot sx, to restore target cluster to sx.
> If a file/dir is
> - renamed, rename it back
> - created in target cluster, delete it
> - modified, put it to the copy list
> - run distcp with the copy list, copy from the source cluster's corresponding
> snapshot
> This could be a new command line switch -rdiff in distcp.
> HDFS-4167 would still be nice to have. It just seems to me that HDFS-9820
> would hopefully be easier to implement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
