[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040134#comment-14040134 ] Hudson commented on HDFS-4667: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1782 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1782/]) HDFS-4667. Capture renamed files/directories in snapshot diff report. Contributed by Jing Zhao and Binglin Chang. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604488) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshotDiffReport.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsSnapshots.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestFullPathNameWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDiffReport.java > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.5.0 > > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, > HDFS-4667.v1.patch, getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040132#comment-14040132 ] Hudson commented on HDFS-4667: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1809 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1809/]) HDFS-4667. Capture renamed files/directories in snapshot diff report. Contributed by Jing Zhao and Binglin Chang. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604488) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshotDiffReport.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsSnapshots.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestFullPathNameWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDiffReport.java > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.5.0 > > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, > HDFS-4667.v1.patch, getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040108#comment-14040108 ] Hudson commented on HDFS-4667: -- FAILURE: Integrated in Hadoop-Yarn-trunk #591 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/591/]) HDFS-4667. Capture renamed files/directories in snapshot diff report. Contributed by Jing Zhao and Binglin Chang. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604488) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshotDiffReport.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsSnapshots.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestFullPathNameWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDiffReport.java > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.5.0 > > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, > HDFS-4667.v1.patch, getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039985#comment-14039985 ] Hudson commented on HDFS-4667: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5750 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5750/]) HDFS-4667. Capture renamed files/directories in snapshot diff report. Contributed by Jing Zhao and Binglin Chang. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604488) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshotDiffReport.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsSnapshots.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestFullPathNameWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDiffReport.java > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.5.0 > > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, > HDFS-4667.v1.patch, getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039667#comment-14039667 ] Binglin Chang commented on HDFS-4667: - bq. Binglin Chang, do you have any other comments? No, thanks for the patch, lgtm > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, > HDFS-4667.v1.patch, getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039491#comment-14039491 ] Jing Zhao commented on HDFS-4667: - Thanks for the review, Nicholas! [~decster], do you have any other comments? > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, > HDFS-4667.v1.patch, getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[
https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038585#comment-14038585
]
Hadoop QA commented on HDFS-4667:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651621/HDFS-4667.004.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7187//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7187//console
This message is automatically generated.
> Capture renamed files/directories in snapshot diff report
> -
>
> Key: HDFS-4667
> URL: https://issues.apache.org/jira/browse/HDFS-4667
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Jing Zhao
>Assignee: Binglin Chang
> Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch,
> HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch,
> HDFS-4667.v1.patch, getfullname-snapshot-support.patch
>
>
> Currently in the diff report we only show file/dir creation, deletion and
> modification. After rename with snapshots is supported, renamed file/dir
> should also be captured in the diff report.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038165#comment-14038165 ] Tsz Wo Nicholas Sze commented on HDFS-4667: --- Patch looks good. Some comments: - Need to document how rename is captured in the diff report. - Add a getRenameEntry(inodeId) to SnapshotDiffInfo. - I suggest move SnapshotDiffInfo out from INodeDirectorySnapshottable since it is getting big. We should also refactor some other snapshot diff report code out. We may do it separately. - The patch does not apply any more. Need to update it. > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.demo.patch, HDFS-4667.v1.patch, > getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032272#comment-14032272 ] Binglin Chang commented on HDFS-4667: - Thanks for the updates [~jingzhao], I will have a look, it may take 1 or 2. > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.demo.patch, HDFS-4667.v1.patch, > getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[
https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14030511#comment-14030511
]
Hadoop QA commented on HDFS-4667:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650248/HDFS-4667.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7109//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7109//console
This message is automatically generated.
> Capture renamed files/directories in snapshot diff report
> -
>
> Key: HDFS-4667
> URL: https://issues.apache.org/jira/browse/HDFS-4667
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Jing Zhao
>Assignee: Binglin Chang
> Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch,
> HDFS-4667.003.patch, HDFS-4667.demo.patch, HDFS-4667.v1.patch,
> getfullname-snapshot-support.patch
>
>
> Currently in the diff report we only show file/dir creation, deletion and
> modification. After rename with snapshots is supported, renamed file/dir
> should also be captured in the diff report.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[
https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028685#comment-14028685
]
Hadoop QA commented on HDFS-4667:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12649916/HDFS-4667.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDistributedFileSystem
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7090//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7090//console
This message is automatically generated.
> Capture renamed files/directories in snapshot diff report
> -
>
> Key: HDFS-4667
> URL: https://issues.apache.org/jira/browse/HDFS-4667
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Jing Zhao
>Assignee: Binglin Chang
> Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch,
> HDFS-4667.demo.patch, HDFS-4667.v1.patch, getfullname-snapshot-support.patch
>
>
> Currently in the diff report we only show file/dir creation, deletion and
> modification. After rename with snapshots is supported, renamed file/dir
> should also be captured in the diff report.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[
https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028479#comment-14028479
]
Hadoop QA commented on HDFS-4667:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12649851/HDFS-4667.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7087//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7087//console
This message is automatically generated.
> Capture renamed files/directories in snapshot diff report
> -
>
> Key: HDFS-4667
> URL: https://issues.apache.org/jira/browse/HDFS-4667
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Jing Zhao
>Assignee: Binglin Chang
> Attachments: HDFS-4667.002.patch, HDFS-4667.demo.patch,
> HDFS-4667.v1.patch, getfullname-snapshot-support.patch
>
>
> Currently in the diff report we only show file/dir creation, deletion and
> modification. After rename with snapshots is supported, renamed file/dir
> should also be captured in the diff report.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819754#comment-13819754 ] Binglin Chang commented on HDFS-4667: - Hi [~jingzhao], what's the status of "get target full path" approach? Could you give me some input maybe I can help on this? > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: HDFS-4667.demo.patch, HDFS-4667.v1.patch, > getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[
https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13750141#comment-13750141
]
Binglin Chang commented on HDFS-4667:
-
And another issue about symlink, currently the original code treat all symlink
delete/create as modify, but symlink is immutable.
{code}
if (cnode.isSymlink() && dnode.isSymlink()) {
dList.add(new DiffReportEntry(DiffType.MODIFY, fullPath));
} else {
{code}
In this patch, symlink with same name but different inodeid are treated as one
CREATE and one DELETE
> Capture renamed files/directories in snapshot diff report
> -
>
> Key: HDFS-4667
> URL: https://issues.apache.org/jira/browse/HDFS-4667
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Jing Zhao
>Assignee: Binglin Chang
> Attachments: getfullname-snapshot-support.patch,
> HDFS-4667.demo.patch, HDFS-4667.v1.patch
>
>
> Currently in the diff report we only show file/dir creation, deletion and
> modification. After rename with snapshots is supported, renamed file/dir
> should also be captured in the diff report.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13750133#comment-13750133 ] Binglin Chang commented on HDFS-4667: - Here is another unsolved issue in the patch: When we only rename a dir(without changing its children), sdir.computeDiffBetweenSnapshots can correctly identify there is no change. When we rename a file without changing other property, file.getDiffs().changedBetweenSnapshots(earlierSnapshot,laterSnapshot) will always assume there is a change, this is the original behavior, is it reasonable? > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: getfullname-snapshot-support.patch, > HDFS-4667.demo.patch, HDFS-4667.v1.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749863#comment-13749863 ] Jing Zhao commented on HDFS-4667: - bq. we need to include both [+ D/D_new] and [R D/c/a -> D/D_new] in diff report Agree. That's why I said "If a file/dir is moved from inside into D_new, we can still follow the reference links of the source to identify the file/dir in D_new" in my previous comment. But here the challenge will still be "to get target full path", and if we cannot, we will have to traverse all the newly-created directories. > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: getfullname-snapshot-support.patch, > HDFS-4667.demo.patch, HDFS-4667.v1.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749859#comment-13749859 ] Binglin Chang commented on HDFS-4667: - bq. Suppose here we created snapshot s0 and s1 under directory D... In most cases we can skip all its sub-items, but suppose we have D/c/a, and we create D_new, and rename D/c/a -> D/D_new/a and delete D/c, we need to include both [+ D/D_new] and [R D/c/a -> D/D_new] in diff report, because some of D_new's children can be moved rather than fully copied. > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: getfullname-snapshot-support.patch, > HDFS-4667.demo.patch, HDFS-4667.v1.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749850#comment-13749850 ] Jing Zhao commented on HDFS-4667: - bq. this is the same effect as traverse all entire snapshottable directory's both versions (from snapshot, to snapshot). Suppose here we created snapshot s0 and s1 under directory D. After creating s0, we create a new directory D/D_new. So when computing the snapshot diff between s0 and s1, after we find that D_new is a newly-created dir after s0, we do not need to traverse D_new any more. In the diff report, we report D_new as created and skip all its sub-items. And if we want to know what files/directories were exactly added under D_new while snapshot s1, we only need to list D/.snapshot/s1/D_new. bq. to be able to get target full path (to check permission and fill DiffEntry with respect to snapshot_to) Yeah, this brings us extra complexity. Looks like in some cases it is even impossible to know the exact full path (e.g., another rename also happens in an ancestor node and there may be no way to know which rename happened first unless we also record DstReference#dstSnapshotId in WithName). Let me spend some more time on this. If we cannot find a way to determine the full path, I think your current solution may be the best way, while using the reference links can act as an extra check to detect Rename among the remaining Deleted entries (and the target of the Rename entries has to be skipped in the report). > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: getfullname-snapshot-support.patch, > HDFS-4667.demo.patch, HDFS-4667.v1.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[
https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749843#comment-13749843
]
Hadoop QA commented on HDFS-4667:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12599867/getfullname-snapshot-support.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4884//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4884//console
This message is automatically generated.
> Capture renamed files/directories in snapshot diff report
> -
>
> Key: HDFS-4667
> URL: https://issues.apache.org/jira/browse/HDFS-4667
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Jing Zhao
>Assignee: Binglin Chang
> Attachments: getfullname-snapshot-support.patch,
> HDFS-4667.demo.patch, HDFS-4667.v1.patch
>
>
> Currently in the diff report we only show file/dir creation, deletion and
> modification. After rename with snapshots is supported, renamed file/dir
> should also be captured in the diff report.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749824#comment-13749824 ] Binglin Chang commented on HDFS-4667: - Yes, you are right, I should probably clarify my comments: All the directories(created/deleted) in the snapshottable directory need to be traversed, this is the same effect as traverse all entire snapshottable directory's both versions (from snapshot, to snapshot). Since D_new is not in either snapshots, we do not need to traverse D_new. bq. but it's better to show the diff type as "Rename" Yes, this is better, but we need to be able to get target full path (to check permission and fill DiffEntry with respect to snapshot_to) first, and snapshot_to is somehow invalid out side the snapshottable directory... so I am not sure this is feasible. > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: getfullname-snapshot-support.patch, > HDFS-4667.demo.patch, HDFS-4667.v1.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749820#comment-13749820 ] Jing Zhao commented on HDFS-4667: - bq. We currently don't check the permissions of inodes outside the snapshottable directory, is it legal to expose those information? If the target of a rename operation is outside of the snapshottable directory, we can also check the corresponding permission. And if the user does not have the read permission, we can hide the target in the diff report. But as the first step, I think hiding target inode information for all "outside rename" scenario is also OK (but it's better to show the diff type as "Rename"). bq. if the source directory is not in this snapshot root, it is better to consider it as newly created, which is easier for distcp to handle. If a new file/directory under the snapshottable dir is actually the target of a rename operation, and the source is outside, in fact it may be impossible for us to identify the source (e.g., the source is not in any snapshot). So I think we should show the file/dir as newly created in this case. bq. I find that all the directories(created/deleted) need to be traversed, because newly-created/deleted directories may contain renamed dirs/files. For a newly-created directory D_new, if a file/dir is moved from outside (of the snapshottable dir) into D_new, we do not need to report the renamed file/dir. If a file/dir is moved from inside into D_new, we can still follow the reference links of the source to identify the file/dir in D_new. And we do not need to report file/dir moved from D_new to somewhere else. Thus we do not need to traverse D_new here. > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: getfullname-snapshot-support.patch, > HDFS-4667.demo.patch, HDFS-4667.v1.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749814#comment-13749814 ] Binglin Chang commented on HDFS-4667: - Thanks for the comments, [~jingzhao]. About rename across snapshottable directory, I have some concerns: 1. We currently don't check the permissions of inodes outside the snapshottable directory, is it legal to expose those information? 2. The diff is most likely to be used for incremental backup(using distcp), in this context, for a rename operation, if the source directory is not in this snapshot root, it is better to consider it as newly created, which is easier for distcp to handle. bq. the rename outside of a snapshottable directory can be detected. Yes, I considered this approach, it can be a better way of generating diff, I tried this, but currently we don't have a method to get an inode's full path given a specified snapshot, e.g. FSDirectory.getFullPathName(INode inode, Snapshot snapshot). I tried to write one, but I am not sure it is correct. Please review the newly attached the patch. bq. Another possible advantage of this method is that we do not need to check the subtree of a new-created directory while generating the diff report. I find that all the directories(created/deleted) need to be traversed, because newly-created/deleted directories may contain renamed dirs/files. > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: HDFS-4667.demo.patch, HDFS-4667.v1.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[
https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749774#comment-13749774
]
Jing Zhao commented on HDFS-4667:
-
Hi Binglin,
Thanks again for working on this jira! So I think the current patch can
nicely report the rename happening within a single snapshottable directory.
However, if we move a file/directory out of a snapshottable directory to
another directory, the current patch will still report the change as DELETE.
This is because the current solution only does comparisons among the
created/deleted items under the same snapshottable directory. A unit test the
current patch may fail is (to be added to TestSnapshotDiffReport):
{code}
public void testDiffReportWithRename() throws Exception {
final Path dir = new Path("/dir");
final Path file0 = new Path(dir, "file0");
final Path file1 = new Path(dir, "file1");
DFSTestUtil.createFile(hdfs, file0, BLOCKSIZE, REPLICATION, seed);
DFSTestUtil.createFile(hdfs, file1, BLOCKSIZE, REPLICATION, seed);
final Path dir2 = new Path("/dir2");
hdfs.mkdirs(dir2);
hdfs.allowSnapshot(dir);
hdfs.createSnapshot(dir, "s0");
// move file0 to dir2
final Path dir2file0 = new Path(dir2, file0.getName());
hdfs.rename(file0, dir2file0, Rename.OVERWRITE);
hdfs.createSnapshot(dir, "s1");
SnapshotDiffReport diffReport = hdfs.getSnapshotDiffReport(dir, "s0", "s1");
List diffEntries = diffReport.getDiffList();
assertEquals(2, diffEntries.size());
assertEquals(DiffType.RENAME, diffEntries.get(1).getType());
}
{code}
In fact, the rename outside of a snapshottable directory can be detected if you
check the inside details of an INodeReference. Currently we add an
INodeReference.WithName instance into the deleted list for the source file/dir
(if the file/dir is contained in a snapshot). The WithName instance is linked
with a WithCount node, which is further linked with a DstReference node. The
DstReference node is the target of the rename operation. If another rename
operation happened on the target file/dir, the DstReference node may be
converted to a WithName node and stored in the WithCount node. So by checking
the corresponding DstReference node and the WithName node list stored in the
WithCount node, we should be able to handle the "outside rename" scenario.
Another possible advantage of this method is that we do not need to check the
subtree of a new-created directory while generating the diff report.
> Capture renamed files/directories in snapshot diff report
> -
>
> Key: HDFS-4667
> URL: https://issues.apache.org/jira/browse/HDFS-4667
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Jing Zhao
>Assignee: Binglin Chang
> Attachments: HDFS-4667.demo.patch, HDFS-4667.v1.patch
>
>
> Currently in the diff report we only show file/dir creation, deletion and
> modification. After rename with snapshots is supported, renamed file/dir
> should also be captured in the diff report.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[
https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749583#comment-13749583
]
Hadoop QA commented on HDFS-4667:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12599804/HDFS-4667.v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4878//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4878//console
This message is automatically generated.
> Capture renamed files/directories in snapshot diff report
> -
>
> Key: HDFS-4667
> URL: https://issues.apache.org/jira/browse/HDFS-4667
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Jing Zhao
>Assignee: Binglin Chang
> Attachments: HDFS-4667.demo.patch, HDFS-4667.v1.patch
>
>
> Currently in the diff report we only show file/dir creation, deletion and
> modification. After rename with snapshots is supported, renamed file/dir
> should also be captured in the diff report.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741433#comment-13741433 ] Jing Zhao commented on HDFS-4667: - Thanks for working on this Binglin! Will check the patch soon. > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: HDFS-4667.demo.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636550#comment-13636550 ] Tsz Wo (Nicholas), SZE commented on HDFS-4667: -- BTW, getSnapshotDiffReport should check read permission for the entire subtrees of the beginning and ending snapshots. > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
