[
https://issues.apache.org/jira/browse/HDFS-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15544058#comment-15544058
]
Yongjun Zhang commented on HDFS-9820:
-------------------------------------
Thanks a lot for your review and comments [~andrew.wang]!
Good comments!
Indeed, the difference between -diff and -rdiff would be very minimum if the
snapshot diff were symmetric. For example, {{getRenameAndDeleteDiffsRdiff}}
would look the same as {{getRenameAndDeleteDiffsFdiff}} when HDFS-10263 is
fixed.
One main sharing between -diff and -rdiff done in the patch is the
infrastructure that handle snapshot diff, that is, where to do the following
steps and to handle the difference between forward and backward snapshot diff
report accordingly:
- calculate snapshot diff
- find out operations to do sync
- do sync (rename/delete)
- create copy listing based on changed items
Will address your other comments soon.
Thanks.
> Improve distcp to support efficient restore to an earlier snapshot
> ------------------------------------------------------------------
>
> Key: HDFS-9820
> URL: https://issues.apache.org/jira/browse/HDFS-9820
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: distcp
> Affects Versions: 2.6.4
> Reporter: Yongjun Zhang
> Assignee: Yongjun Zhang
> Attachments: HDFS-9820.001.patch, HDFS-9820.002.patch,
> HDFS-9820.003.patch, HDFS-9820.004.patch, HDFS-9820.005.patch
>
>
> A common use scenario (scenaio 1):
> # create snapshot sx in clusterX,
> # do some experiemnts in clusterX, which creates some files.
> # throw away the files changed and go back to sx.
> Another scenario (scenario 2) is, there is a production cluster and a backup
> cluster, we periodically sync up the data from production cluster to the
> backup cluster with distcp.
> The cluster in scenario 1 could be the backup cluster in scenario 2.
> For scenario 1:
> HDFS-4167 intends to restore HDFS to the most recent snapshot, and there are
> some complexity and challenges. Before that jira is implemented, we count on
> distcp to copy from snapshot to the current state. However, the performance
> of this operation could be very bad because we have to go through all files
> even if we only changed a few files.
> For scenario 2:
> HDFS-7535 improved distcp performance by avoiding copying files that changed
> name since last backup.
> On top of HDFS-7535, HDFS-8828 improved distcp performance when copying data
> from source to target cluster, by only copying changed files since last
> backup. The way it works is use snapshot diff to find out all files changed,
> and copy the changed files only.
> See
> https://blog.cloudera.com/blog/2015/12/distcp-performance-improvements-in-apache-hadoop/
> This jira is to propose a variation of HDFS-8828, to find out the files
> changed in target cluster since last snapshot sx, and copy these from
> snapshot sx of either the source or the target cluster, to restore target
> cluster's current state to sx.
> Specifically,
> If a file/dir is
> - renamed, rename it back
> - created in target cluster, delete it
> - modified, put it to the copy list
> - run distcp with the copy list, copy from the source cluster's corresponding
> snapshot
> This could be a new command line switch -rdiff in distcp.
> As a native restore feature, HDFS-4167 would still be ideal to have. However,
> HDFS-9820 would hopefully be easier to implement, before HDFS-4167 is in
> place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
