[jira] [Commented] (HDFS-6266) Identify full path for a given INode
[ https://issues.apache.org/jira/browse/HDFS-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980926#comment-13980926 ] Hudson commented on HDFS-6266: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #551 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/551/]) HDFS-6266. Identify full path for a given INode. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1589920) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestFullPathNameWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java Identify full path for a given INode Key: HDFS-6266 URL: https://issues.apache.org/jira/browse/HDFS-6266 Project: Hadoop HDFS Issue Type: Improvement Components: snapshots Reporter: Jing Zhao Assignee: Jing Zhao Fix For: 2.5.0 Attachments: HDFS-6266.000.patch, HDFS-6266.001.patch Currently when identifying the full path of a given inode, FSDirectory#getPathComponents and FSDirectory#getFullPathName can only handle normal cases where the inode and its ancestors are not in any snapshot. This jira aims to provide support to handle snapshots. This can be useful for identifying the Rename change in a snapshot diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6266) Identify full path for a given INode
[ https://issues.apache.org/jira/browse/HDFS-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980989#comment-13980989 ] Hudson commented on HDFS-6266: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1768 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1768/]) HDFS-6266. Identify full path for a given INode. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1589920) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestFullPathNameWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java Identify full path for a given INode Key: HDFS-6266 URL: https://issues.apache.org/jira/browse/HDFS-6266 Project: Hadoop HDFS Issue Type: Improvement Components: snapshots Reporter: Jing Zhao Assignee: Jing Zhao Fix For: 2.5.0 Attachments: HDFS-6266.000.patch, HDFS-6266.001.patch Currently when identifying the full path of a given inode, FSDirectory#getPathComponents and FSDirectory#getFullPathName can only handle normal cases where the inode and its ancestors are not in any snapshot. This jira aims to provide support to handle snapshots. This can be useful for identifying the Rename change in a snapshot diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6266) Identify full path for a given INode
[ https://issues.apache.org/jira/browse/HDFS-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981020#comment-13981020 ] Hudson commented on HDFS-6266: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1742 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1742/]) HDFS-6266. Identify full path for a given INode. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1589920) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestFullPathNameWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java Identify full path for a given INode Key: HDFS-6266 URL: https://issues.apache.org/jira/browse/HDFS-6266 Project: Hadoop HDFS Issue Type: Improvement Components: snapshots Reporter: Jing Zhao Assignee: Jing Zhao Fix For: 2.5.0 Attachments: HDFS-6266.000.patch, HDFS-6266.001.patch Currently when identifying the full path of a given inode, FSDirectory#getPathComponents and FSDirectory#getFullPathName can only handle normal cases where the inode and its ancestors are not in any snapshot. This jira aims to provide support to handle snapshots. This can be useful for identifying the Rename change in a snapshot diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6266) Identify full path for a given INode
[
https://issues.apache.org/jira/browse/HDFS-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980379#comment-13980379
]
Hadoop QA commented on HDFS-6266:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12641769/HDFS-6266.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6723//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6723//console
This message is automatically generated.
Identify full path for a given INode
Key: HDFS-6266
URL: https://issues.apache.org/jira/browse/HDFS-6266
Project: Hadoop HDFS
Issue Type: Improvement
Components: snapshots
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-6266.000.patch, HDFS-6266.001.patch
Currently when identifying the full path of a given inode,
FSDirectory#getPathComponents and FSDirectory#getFullPathName can only handle
normal cases where the inode and its ancestors are not in any snapshot. This
jira aims to provide support to handle snapshots. This can be useful for
identifying the Rename change in a snapshot diff report.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6266) Identify full path for a given INode
[ https://issues.apache.org/jira/browse/HDFS-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980634#comment-13980634 ] Hudson commented on HDFS-6266: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5568 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5568/]) HDFS-6266. Identify full path for a given INode. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1589920) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestFullPathNameWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java Identify full path for a given INode Key: HDFS-6266 URL: https://issues.apache.org/jira/browse/HDFS-6266 Project: Hadoop HDFS Issue Type: Improvement Components: snapshots Reporter: Jing Zhao Assignee: Jing Zhao Fix For: 2.5.0 Attachments: HDFS-6266.000.patch, HDFS-6266.001.patch Currently when identifying the full path of a given inode, FSDirectory#getPathComponents and FSDirectory#getFullPathName can only handle normal cases where the inode and its ancestors are not in any snapshot. This jira aims to provide support to handle snapshots. This can be useful for identifying the Rename change in a snapshot diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6266) Identify full path for a given INode
[
https://issues.apache.org/jira/browse/HDFS-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13979219#comment-13979219
]
Tsz Wo Nicholas Sze commented on HDFS-6266:
---
Question: Suppose there is a file /dir/foo, where dir is snapshottable. Then,
create snapshot s1, s2 and s3 so that foo is in all three snapshots. Now
delete foo and then create snapshot s4. If we call
getPathComponentsWithSnapshot(foo), it will return /dir/.snapshot/s3/foo. Why
not returning /dir/.snapshot/s1/foo (or s2)? Would it work for computing
snapshot diff between s4 and s2?
Other comments
- In the code below, even if sid is CURRENT_STATE_ID, inSnapshot will be set to
true. I guess it probably is not a bug since you have tests covering it.
Maybe we should rename the variable inSnapshot? Or, could we add sid !=
Snapshot.CURRENT_STATE_ID to the if-statement below?
{code}
int sid = parent.asDirectory().searchChild(inode);
Preconditions.checkState(sid != Snapshot.NO_SNAPSHOT_ID);
if (snapshotId == Snapshot.CURRENT_STATE_ID) {
snapshotId = sid;
inSnapshot = true;
}
{code}
- We should add snapshotId != Snapshot.CURRENT_STATE_ID to the if-statement
below.
{code}
if (inode instanceof INodeDirectory
inode.asDirectory().isSnapshottable() inSnapshot) {
...
}
{code}
- getPathComponentsWithSnapshot should be static
- findINodeInSnapshot actaully is finding snapshot, how about renaming it to
something like findSnapshotDeleted?
Identify full path for a given INode
Key: HDFS-6266
URL: https://issues.apache.org/jira/browse/HDFS-6266
Project: Hadoop HDFS
Issue Type: Improvement
Components: snapshots
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-6266.000.patch
Currently when identifying the full path of a given inode,
FSDirectory#getPathComponents and FSDirectory#getFullPathName can only handle
normal cases where the inode and its ancestors are not in any snapshot. This
jira aims to provide support to handle snapshots. This can be useful for
identifying the Rename change in a snapshot diff report.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6266) Identify full path for a given INode
[ https://issues.apache.org/jira/browse/HDFS-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976721#comment-13976721 ] Daryn Sharp commented on HDFS-6266: --- I'm only moderately familiar with snapshot handling, but how is a deleted file inode that belongs to multiple snapshots handled? Ie. which snapshot will it appear to belong to? Is the need for this patch due to loss of the original array of inodes found during path resolution? Identify full path for a given INode Key: HDFS-6266 URL: https://issues.apache.org/jira/browse/HDFS-6266 Project: Hadoop HDFS Issue Type: Improvement Components: snapshots Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-6266.000.patch Currently when identifying the full path of a given inode, FSDirectory#getPathComponents and FSDirectory#getFullPathName can only handle normal cases where the inode and its ancestors are not in any snapshot. This jira aims to provide support to handle snapshots. This can be useful for identifying the Rename change in a snapshot diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6266) Identify full path for a given INode
[ https://issues.apache.org/jira/browse/HDFS-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977079#comment-13977079 ] Jing Zhao commented on HDFS-6266: - Thanks for the comments, [~daryn]. bq. how is a deleted file inode that belongs to multiple snapshots handled? The current patch uses the latest snapshot in the path. For e.g., suppose we have created snapshots s1, s2, and s3 for directory /foo. If we create file /foo/bar after s1 but before s2, and delete bar after s3, bar now belongs to snapshot s2 and s3, and the inode for bar is stored in the deleted list of s3. The current patch will return /foo/.snapshot/s3/bar as the full path for bar. If we later delete snapshot s3, /foo/.snapshot/s2/bar will be returned. bq. Is the need for this patch due to loss of the original array of inodes found during path resolution? The main use case for this patch currently is for snapshot diff report. The current snapshot diff report functionality cannot identify rename. For a renamed file/directory, we can use information stored in INodeReference to find the target inode in the rename operation. Then we can use this patch to identify the full path of the inode and show it in the diff report. Identify full path for a given INode Key: HDFS-6266 URL: https://issues.apache.org/jira/browse/HDFS-6266 Project: Hadoop HDFS Issue Type: Improvement Components: snapshots Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-6266.000.patch Currently when identifying the full path of a given inode, FSDirectory#getPathComponents and FSDirectory#getFullPathName can only handle normal cases where the inode and its ancestors are not in any snapshot. This jira aims to provide support to handle snapshots. This can be useful for identifying the Rename change in a snapshot diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6266) Identify full path for a given INode
[
https://issues.apache.org/jira/browse/HDFS-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977327#comment-13977327
]
Hadoop QA commented on HDFS-6266:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12641161/HDFS-6266.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6691//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6691//console
This message is automatically generated.
Identify full path for a given INode
Key: HDFS-6266
URL: https://issues.apache.org/jira/browse/HDFS-6266
Project: Hadoop HDFS
Issue Type: Improvement
Components: snapshots
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-6266.000.patch
Currently when identifying the full path of a given inode,
FSDirectory#getPathComponents and FSDirectory#getFullPathName can only handle
normal cases where the inode and its ancestors are not in any snapshot. This
jira aims to provide support to handle snapshots. This can be useful for
identifying the Rename change in a snapshot diff report.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
