[jira] [Commented] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646555#comment-13646555 ] Hudson commented on HDFS-4758: -- Integrated in Hadoop-Hdfs-Snapshots-Branch-build #174 (See [https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/174/]) HDFS-4758. Disallow nested snapshottable directories and unwrap RemoteException. (Revision 1477867) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1477867 Files : * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-2802.txt * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestNestedSnapshots.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshot.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDiffReport.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotMetrics.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotRename.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshottableDirListing.java Disallow nested snapshottable directories - Key: HDFS-4758 URL: https://issues.apache.org/jira/browse/HDFS-4758 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: Snapshot (HDFS-2802) Attachments: h4758_20140426.patch, h4758_20140429b.patch, h4758_20140429.patch, h4758_20140430.patch Nested snapshottable directories are supported by the current implementation. However, it seems that there are no good use cases for nested snapshottable directories. So we disable it for now until someone has a valid use case for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646073#comment-13646073 ] Jing Zhao commented on HDFS-4758: - The patch looks good to me. +1 for the patch. Disallow nested snapshottable directories - Key: HDFS-4758 URL: https://issues.apache.org/jira/browse/HDFS-4758 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h4758_20140426.patch, h4758_20140429b.patch, h4758_20140429.patch, h4758_20140430.patch Nested snapshottable directories are supported by the current implementation. However, it seems that there are no good use cases for nested snapshottable directories. So we disable it for now until someone has a valid use case for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643429#comment-13643429 ] Tsz Wo (Nicholas), SZE commented on HDFS-4758: -- A common use case for snapshots is a short-lived snapshot which is then used as the source for distcp. ... This does not seem to related to nested snapshots. ... user might have different backup policies for /user (once every day) and /user/hive (every 8 hrs) ... In such case, admin may take a snapshot for all the subdirs in /user once per day. If the current restrictions takes away some use cases, so be it. Lets turn it back on later if we cannot live without it. Agree. We can turn it back if necessary. Disallow nested snapshottable directories - Key: HDFS-4758 URL: https://issues.apache.org/jira/browse/HDFS-4758 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Nested snapshottable directories are supported by the current implementation. However, it seems that there are no good use cases for nested snapshottable directories. So we disable it for now until someone has a valid use case for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642234#comment-13642234 ] Todd Lipcon commented on HDFS-4758: --- Based on the discussion at the contributors meetup a few months back, didn't a lot of us anticipate the use case of a snapshottable /user, and then allowing each user or application to then make snapshots of their own /user/foo? Or a snapshottable root directory and then snapshottable subdirs? Disallow nested snapshottable directories - Key: HDFS-4758 URL: https://issues.apache.org/jira/browse/HDFS-4758 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Nested snapshottable directories are supported by the current implementation. However, it seems that there are no good use cases for nested snapshottable directories. So we disable it for now until someone has a valid use case for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642244#comment-13642244 ] Tsz Wo (Nicholas), SZE commented on HDFS-4758: -- For your first example, we may set snapshottable for all users. Then taking snapshots of /user can be done by taking snapshots for all user dirs. For root, I think we should not encourage taking snapshots since all the tmp files will sit in a snapshot forever. Instead, we should encourage taking sanpshots in subdirectories. It seems that nested snapshottable directories bring only a little convenience with a lot of inefficiencies. What do you think? Disallow nested snapshottable directories - Key: HDFS-4758 URL: https://issues.apache.org/jira/browse/HDFS-4758 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Nested snapshottable directories are supported by the current implementation. However, it seems that there are no good use cases for nested snapshottable directories. So we disable it for now until someone has a valid use case for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642256#comment-13642256 ] Todd Lipcon commented on HDFS-4758: --- bq. For your first example, we may set snapshottable for all users. Then taking snapshots of /user can be done by taking snapshots for all user dirs. But then the snapshot is not atomic across users, which makes it difficult to do a point in time backup of a cluster by snapshot+distcp bq. For root, I think we should not encourage taking snapshots since all the tmp files will sit in a snapshot forever. Instead, we should encourage taking sanpshots in subdirectories. A common use case for snapshots is a short-lived snapshot which is then used as the source for distcp. The distcp can explicitly exclude copying tmp files. Once the distcp is complete, then the snapshot can be removed, so the space usage of the tmp files is only temporary. Disallow nested snapshottable directories - Key: HDFS-4758 URL: https://issues.apache.org/jira/browse/HDFS-4758 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Nested snapshottable directories are supported by the current implementation. However, it seems that there are no good use cases for nested snapshottable directories. So we disable it for now until someone has a valid use case for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642270#comment-13642270 ] Hari Mankude commented on HDFS-4758: Actually one use case of nested snapshots that I see is that user might have different backup policies for /user (once every day) and /user/hive (every 8 hrs). When backing up /user, it is possible to setup exclusions of /user/hive directory so that two copies of /user/hive is not made. However, if snapshots cannot be taken of /user and /user/hive at the same time, it would be a disadvantage. Disallow nested snapshottable directories - Key: HDFS-4758 URL: https://issues.apache.org/jira/browse/HDFS-4758 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Nested snapshottable directories are supported by the current implementation. However, it seems that there are no good use cases for nested snapshottable directories. So we disable it for now until someone has a valid use case for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642280#comment-13642280 ] Suresh Srinivas commented on HDFS-4758: --- The current branch does support nested snapshots. The reason why this is being proposed is to provide limited functionality early on and if that suffices remove complexity associated with supporting and testing nested snapshots at arbitrary levels. If we support nested snapshots now, it will not be possible to take it away in the future. In my discussion with Jing and Nicholas we had considered the following: # Allow snapshot of / as a special case and that is the only snapshottable directory with in which another snapshottable directory is allowed. # Disallow nested snapshots in the first phase. See how how this is used by the users and enable it either as stated in the first bullet or remove the restriction altogether. If the current restrictions takes away some use cases, so be it. Lets turn it back on later if we cannot live without it. Disallow nested snapshottable directories - Key: HDFS-4758 URL: https://issues.apache.org/jira/browse/HDFS-4758 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Nested snapshottable directories are supported by the current implementation. However, it seems that there are no good use cases for nested snapshottable directories. So we disable it for now until someone has a valid use case for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642316#comment-13642316 ] Hari Mankude commented on HDFS-4758: The trade-off is between usability vs complexity. In this case, it might result in issues where a user has taken a snapshot of /user/foo/dir1 and admin finds that system-wide snaps cannot be taken at say /user dir levels since there are several users with their snapshots at lower directories. This might limit the usability of the feature. Disallow nested snapshottable directories - Key: HDFS-4758 URL: https://issues.apache.org/jira/browse/HDFS-4758 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Nested snapshottable directories are supported by the current implementation. However, it seems that there are no good use cases for nested snapshottable directories. So we disable it for now until someone has a valid use case for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642328#comment-13642328 ] Jing Zhao commented on HDFS-4758: - bq. it might result in issues where a user has taken a snapshot of /user/foo/dir1 and admin finds that system-wide snaps cannot be taken at say /user dir levels since there are several users with their snapshots at lower directories. This may not be an issue because only admin can allow snapshot on a directory (i.e., to convert a directory to a snapshottable directory). The admin can specify a set of snapshottable directories in the beginning under which no more snapshottable directory will be allowed in the future. Disallow nested snapshottable directories - Key: HDFS-4758 URL: https://issues.apache.org/jira/browse/HDFS-4758 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Nested snapshottable directories are supported by the current implementation. However, it seems that there are no good use cases for nested snapshottable directories. So we disable it for now until someone has a valid use case for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4758) Disallow nested snapshottable directories
[ https://issues.apache.org/jira/browse/HDFS-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642352#comment-13642352 ] Tsz Wo (Nicholas), SZE commented on HDFS-4758: -- But then the snapshot is not atomic across users, ... It is easy to change the API to support it. Disallow nested snapshottable directories - Key: HDFS-4758 URL: https://issues.apache.org/jira/browse/HDFS-4758 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Nested snapshottable directories are supported by the current implementation. However, it seems that there are no good use cases for nested snapshottable directories. So we disable it for now until someone has a valid use case for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
