[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989946#comment-13989946 ] Yongjun Zhang commented on HDFS-6165: - HI [~daryn], I looked into callers of checkPermission, there are two other places in additional to delete operation: - FSNamesystem.getContentSummary method calls checkPermission with FsAction.READ_EXECUTE passed to subAccess. - FSNamesystem.checkSubtreeReadPermission method calls checkPermission with FsAction.READ passed to subAccess. so it looks like that we do need the additional parameter. About the RemoteException, thanks for pointing out that FsShell won't work correctly with other filesystems with the patch. Since there are so many filesystems, the scope of change to address mkdir issue will be much more wide. I'm thinking about handling this in a separate JIRA. What do you guys think? With rmr fixed, it can serve as a workaround for rmdir issue. Thanks. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch, > HDFS-6165.005.patch, HDFS-6165.006.patch, HDFS-6165.006.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11 > /user/userabc/foo-file > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file > Deleted /user/userabc/foo-file > {code} > Using "hdfs dfs -rmdir" command: > {code} > bash-4.1$ hadoop fs -lsr / > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - hdfs supergroup
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987870#comment-13987870 ] Yongjun Zhang commented on HDFS-6165: - Hi [~daryn], Thanks a lot for your comments. Adding the additional parameter is to avoid changing the behavior of any other callers to checkPermission. If subAccess is only used by deleteInternal method, then we can actually remove the additional parameter and change the behavior when we check subAccess. About catching RemoteException, thanks for pointing that out, I will do some further study how to address that. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch, > HDFS-6165.005.patch, HDFS-6165.006.patch, HDFS-6165.006.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11 > /user/userabc/foo-file > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file > Deleted /user/userabc/foo-file > {code} > Using "hdfs dfs -rmdir" command: > {code} > bash-4.1$ hadoop fs -lsr / > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:29 /user > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:28 /user/hdfs > drwxr-xr-x - usrabc users 0 2014-03-28 23:39 /user/usrabc > drwxr-xr-x - abcabc 0 2014-03-28 23:39 > /user/usrabc/foo-empty1 > [root@vm01 usrabc]# su usrabc > [usrabc@vm01 ~]$ hdfs dfs -rmdir /user/usrabc/foo-empty1
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987766#comment-13987766 ] Daryn Sharp commented on HDFS-6165: --- I'm struggling to catch up on jiras so could you post a short summary of why changing the behavior of subaccess is insufficient? It appears to maybe only be used by delete so adding another parameter to permission checking might be overkill. Also, as previously mentioned you can't catch RemoteException since it's specific to hdfs and thus FsShell won't work correctly with other filesystems. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch, > HDFS-6165.005.patch, HDFS-6165.006.patch, HDFS-6165.006.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11 > /user/userabc/foo-file > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file > Deleted /user/userabc/foo-file > {code} > Using "hdfs dfs -rmdir" command: > {code} > bash-4.1$ hadoop fs -lsr / > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:29 /user > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:28 /user/hdfs > drwxr-xr-x - usrabc users 0 2014-03-28 23:39 /user/usrabc > drwxr-xr-x - abcabc 0 2014-03-28 23:39 > /user/usrabc/foo-empty1 > [root@vm01 usrabc]# su usrabc > [usrabc@vm01 ~]$ hdfs dfs -rmdir /user/usrabc/foo-empty1 > rmdir: Permission denied: u
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985972#comment-13985972 ] Yongjun Zhang commented on HDFS-6165: - The failed test appears to be https://issues.apache.org/jira/browse/HADOOP-10062. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch, > HDFS-6165.005.patch, HDFS-6165.006.patch, HDFS-6165.006.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11 > /user/userabc/foo-file > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file > Deleted /user/userabc/foo-file > {code} > Using "hdfs dfs -rmdir" command: > {code} > bash-4.1$ hadoop fs -lsr / > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:29 /user > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:28 /user/hdfs > drwxr-xr-x - usrabc users 0 2014-03-28 23:39 /user/usrabc > drwxr-xr-x - abcabc 0 2014-03-28 23:39 > /user/usrabc/foo-empty1 > [root@vm01 usrabc]# su usrabc > [usrabc@vm01 ~]$ hdfs dfs -rmdir /user/usrabc/foo-empty1 > rmdir: Permission denied: user=usrabc, access=ALL, > inode="/user/usrabc/foo-empty1":abc:abc:drwxr-xr-x > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985954#comment-13985954 ] Hadoop QA commented on HDFS-6165: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642655/HDFS-6165.006.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6777//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6777//console This message is automatically generated. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch, > HDFS-6165.005.patch, HDFS-6165.006.patch, HDFS-6165.006.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985606#comment-13985606 ] Yongjun Zhang commented on HDFS-6165: - Somehow test was not triggered by previous upload, uploaded the same version again to trigger test. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch, > HDFS-6165.005.patch, HDFS-6165.006.patch, HDFS-6165.006.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11 > /user/userabc/foo-file > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file > Deleted /user/userabc/foo-file > {code} > Using "hdfs dfs -rmdir" command: > {code} > bash-4.1$ hadoop fs -lsr / > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:29 /user > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:28 /user/hdfs > drwxr-xr-x - usrabc users 0 2014-03-28 23:39 /user/usrabc > drwxr-xr-x - abcabc 0 2014-03-28 23:39 > /user/usrabc/foo-empty1 > [root@vm01 usrabc]# su usrabc > [usrabc@vm01 ~]$ hdfs dfs -rmdir /user/usrabc/foo-empty1 > rmdir: Permission denied: user=usrabc, access=ALL, > inode="/user/usrabc/foo-empty1":abc:abc:drwxr-xr-x > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984008#comment-13984008 ] Hadoop QA commented on HDFS-6165: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642397/HDFS-6165.005.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6763//console This message is automatically generated. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch, > HDFS-6165.005.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11 > /user/userabc/foo-file > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file > Deleted /user/userabc/foo-file > {code} > Using "hdfs dfs -rmdir" command: > {code} > bash-4.1$ hadoop fs -lsr / > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:29 /user > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:28 /user/hdfs > drwxr-xr-x - usrabc users 0 2014-03-28 23:39 /user/usrabc > drwxr-xr-x - abcabc 0 2014-03-28 23:39 >
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983980#comment-13983980 ] Yongjun Zhang commented on HDFS-6165: - Hi, Thanks a lot for your earlier comments, and thanks Andrew a lot for the detailed review! I just updated patch version 005 to address all. For rmdir, it's the solution I described in above; For rmr solution, I actually did {code} void checkPermission(String path, INodeDirectory root, boolean doCheckOwner, FsAction ancestorAccess, FsAction parentAccess, FsAction access, FsAction subAccess, boolean ignoreEmptyDir, boolean resolveLink) {code} The two parameters "subAccess" and "ignoreEmptyDir" work together, - if subAccess is not NULL, access permission of subDirs are checked, - when subAccess is checked, if ignoreEmptyDir is true, ignore empty directories. To address Andrew's comments {quote} I think the semantics for a recursive delete via DistributedFileSystem#delete are still not quite right. The change you made will work for the shell since it does its own recursion, but we need to do the same "remove if empty dir with read" when recursing via recursive = true too. You might be able to do this by modifying FSPermissionChecker#checkSubAccess appropriately, but a new flag or new code would be safer. {quote} Thanks a lot for pointing this out, indeed it's a problem there. See above described solution, except we agreed that we don't need to check permission for empty dir. {quote} isDirectory, can we add per-parameter javadoc rather than stacking on the @return? I think renaming empty to isEmpty would also help. Nit, also need a space in the ternary empty? and dir.isEmptyDirectory(src)?. {quote} These are now gone with new solution. {code} In Delete, I think it's a bit cleaner to do an instanceof PathIsNotEmptyDirectoryException.class check instead. {code} This is handled in a better way now. I discovered a bug HADOOP-10543 (and posted a patch) when looking at this. With HADOOP-10543 committed, I would be able to do exactly what Andrew suggested. But I think what I have in this new revision should fine too. {quote} Some lines longer than 80 chars {quote} Hopefully all addressed:-) {quote} TestFsShellPrivilege: I gave this a quick pass, but overall it may be better to rewrite these to use the DFS API instead of the shell. We need to test recursive delete, which the shell doesn't do, and we don't really have any shell changes in the latest rev, which lessens the importance of having new shell tests. {quote} I think adding a test infra like what I added give another option here, hopefully the new revision looks better:-) {quote} execCmd needs to do some try/finally to close and restore the streams if there's an exception. Also an extra commented line there. {quote} This FsShell actually took care of catching exception, so the stream will not get lost. Extra comment line removed. {quote} Could we rename this file to "TestFsShellPermission"? Permission is a more standard term. {quote} Done. {quote} This file also should not be in hadoop-tools, but rather hadoop-common. {quote} Because it uses MiniDFSCluster, it can not be in hadoop-common. But I moved to hdfs test area now. {quote} This does a lot of starting and stopping of a MiniCluster for running single-line tests. Can we combine these into a single test? We also don't need any DNs for this cluster, since we're just testing perms. {quote} I refactored the code to take care of this. Since we create file, I still keep DNs. {quote} We have FileSystemTestHelper#createFile for creating files, can save some code Use of @Before and @After blocks might also clarify what's going on. This also should be a JUnit4 test with @Test annotations, not JUnit3. USER_UGI should not be all caps, it's not static final It's a bit ugly how we pass UNEXPECTED_RESULT in for a lot of tests. Can we just pass a boolean for expectSuccess or expectFailure, or maybe a String that we can call assertExceptionContains on? {quote} All are taken care of, except I forgot @before and @After, but hoopefully it looks much better now. {quote} FileEntry looks basically like a FileStatus, can we just use that instead? {quote} FileEntry only have the fields needed for this test, and it's easier to manage in test area. I'm worried using FileStatus would be not easy to control. So I didn't do this. Hope it's acceptable. Thanks in advance for a further review. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13982001#comment-13982001 ] Yongjun Zhang commented on HDFS-6165: - I need to amend the rmr section in my last update. Command: "hdfs dfs -rm -r " Expected result: recursively delete all components under and itself. >From POSIX quote from our earlier discussion, for any component to be deleted, if the component is a file or empty-dir, permissions required are: - Search permission of all components of the path prefix, - WRITE permission of the parent directory of the directory to be removed. if the component to delete is a non-empty-dir, recursively apply the above permissions to its child components. The PROBLEM we are solving here is, that currently FsActions.ALL is passed to subAccess parameter: {code} void checkPermission(String path, INodeDirectory root, boolean doCheckOwner, FsAction ancestorAccess, FsAction parentAccess, FsAction access, FsAction subAccess, boolean resolveLink) {code} which disallows removal of file and empty-dir component under the (got exception asking for ALL permission), if the component doesn't have ALL permissions.. Given an example how to solve this, "hdfs dfs -rm -r ", assume the contains: {code} a a/emptyDir a/nonEmptyDir a/nonEmptyDir/filex {code} to delete , we need the following permissions (ALL means RWX): {code} Prefix components of : search permission (READ and EXECUTE) Parent componet of : ALL : ALL a: ALL a/emptyDir:NONE a/nonEmptyDir: ALL a/nonEmptyDir/filex: NONE {code} So what we can do is to modify checkPermission. Since checkPermission used by multiple operations, the safe change for delete operation is to 1. add a parameter leafAccess: {code} void checkPermission(String path, INodeDirectory root, boolean doCheckOwner, FsAction ancestorAccess, FsAction parentAccess, FsAction access, FsAction subAccess, FsAction leafAccess, boolean resolveLink) {code} Notice that I changed "emptyDirSubAccess" of my earlier version to "leafAccess" to cover both file and dir which are leaf (To be a leaf, the component has to be either a file or an empty dir). And we pass null to "leafAccess" parameter in deleteInternal function. 2. modify checkPermission to apply subAccess on non-leaf component, and leafAccess to leaf component. This change is only applicable to "delete" operation (by deleteInternal method), since checkPermission function is shared across the board, need to examine how to make this change not to impact other operations. Thanks. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > d
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981505#comment-13981505 ] Yongjun Zhang commented on HDFS-6165: - HI Guys, Thanks a lot for the review/comments you did, and sorry again for getting back late. Instead of posting another patch to address all comments, I'd like to summary the solution here first. Sorry for a long posting again. Would you please help review the proposed solution below and see if you agree? I will address other comments when putting together a new revision of patch. Many thanks. There are two commands that have problem here 1. hdfs dfs -rmdir (refer to as rmdir in later discussion) 2. hdfs dfs -rm -r (refer to as rmr in later discussion) Both commands eventually will call FSnamesystem#deleteInternal method {code} private boolean deleteInternal(String src, boolean recursive, boolean enforcePermission, boolean logRetryCache) throws AccessControlException, SafeModeException, UnresolvedLinkException, IOException { {code} The deleteInternal methods throws exception if recursive is not true and the src to be deleted is not empty; otherwise, it will check "necessary" permissions and collect all blocks/inodes to be deleted and delete them recursively. The deletion process excludes snapshottable dirs that has at least one snapshot. Right now it requires FULL permission of the subdirs or files under the target dir to be deleted. This permission checking is also recursive, it requires all child has FULL permission), This is the place we try to fix for different scenarios. rmr calls with the "recursive" parameter passed as true, and rmdir calls with the "recursive" parameter with false. Solution summary: 1. rmdir The recursion issue in the comments you guys made is only relevant to rmr. So the solution for rmdir is a simple: - for nonempty directory, deleteInternal simply throws nonemptydir exception if it's nonempty, and the FsShell side catch the exception - for empty directory, only check parent/prefix permission, ignore the target dir's permission (posix compliant), delete if the permission is satisfied, throw exception otherwise. 2. rmr The last patch (version 004) I posted only checks whether the target dir to be deleted has READ permission (the earlier versions ignore the target dir's permission when it's empty), and I didn't change the behaviour to check subdir for non-empty target dir. For non-empty target dir, the current implementation requires FULL permission of subdir in order to delete a subdir even if the subdir is empty. This is not quite right as [~andrew.wang] pointed out. I'd like to try to implement what Andrew suggested with an additional/different parameter than subAccess of FSPermissionChecker#checkSubAccess E.g. add emptyDirSubAccess {code} void checkPermission(String path, INodeDirectory root, boolean doCheckOwner, FsAction ancestorAccess, FsAction parentAccess, FsAction access, FsAction subAccess, FsAction emptyDirSubAccess, boolean resolveLink) {code} The subAccess will passed FsAction.ALL (as is currently) and emptyDirSubAccess will be passed FsAction.NONE. And if a subdir is not empty, check against subAccess, if it's empty, check against emptyDirSubAccess. About using FsAction.ALL for subAccess parameter, it's a bit over stringent for intemediate path, say, if we want to delete /a/b/c, we don't have to have WRITE permission of /a, but we need to have WRITE permission of /a/b. We might address this issue in a separate JIRA if you agree. Hi [~daryn], the following is actually not true to me {quote} I bet for the described problem, you could delete the no-perms dir if you created a new directory, moved the no-perms dir into it, and then recursively deleted the new directory. {quote} Because the current implementation recursively requires FULL permission of subdirs/files to delete them. If we suddenly change the implementation to allow deleting non-empty dir without checking the subdir/file permission, I'm worried about bad user impact. Thanks again for your time. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directo
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13975064#comment-13975064 ] Yongjun Zhang commented on HDFS-6165: - Hello [~daryn] and [~cnauroth], Thanks a lot for the comments you guys made above! I'm sorry for being late, I have been working on some more priority issues and putting aside this jira, I didn't even see your comments until now. Will address all your comments next week. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11 > /user/userabc/foo-file > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file > Deleted /user/userabc/foo-file > {code} > Using "hdfs dfs -rmdir" command: > {code} > bash-4.1$ hadoop fs -lsr / > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:29 /user > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:28 /user/hdfs > drwxr-xr-x - usrabc users 0 2014-03-28 23:39 /user/usrabc > drwxr-xr-x - abcabc 0 2014-03-28 23:39 > /user/usrabc/foo-empty1 > [root@vm01 usrabc]# su usrabc > [usrabc@vm01 ~]$ hdfs dfs -rmdir /user/usrabc/foo-empty1 > rmdir: Permission denied: user=usrabc, access=ALL, > inode="/user/usrabc/foo-empty1":abc:abc:drwxr-xr-x > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13970062#comment-13970062 ] Chris Nauroth commented on HDFS-6165: - bq. I think the only change, for now, is the NN should only check permissions of the parent directory for deletes. +1 to this proposal. I think it's the simplest approach to fixing the current bug. Enhancements beyond that could be addressed in separate patches if needed. [~yzhangal], thank you for sticking with this tricky issue. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11 > /user/userabc/foo-file > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file > Deleted /user/userabc/foo-file > {code} > Using "hdfs dfs -rmdir" command: > {code} > bash-4.1$ hadoop fs -lsr / > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:29 /user > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:28 /user/hdfs > drwxr-xr-x - usrabc users 0 2014-03-28 23:39 /user/usrabc > drwxr-xr-x - abcabc 0 2014-03-28 23:39 > /user/usrabc/foo-empty1 > [root@vm01 usrabc]# su usrabc > [usrabc@vm01 ~]$ hdfs dfs -rmdir /user/usrabc/foo-empty1 > rmdir: Permission denied: user=usrabc, access=ALL, > inode="/user/usrabc/foo-empty1":abc:abc:drwxr-xr-x > {code} -- This message was sent by Atlas
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13970052#comment-13970052 ] Daryn Sharp commented on HDFS-6165: --- After looking over the FsShell code, I'll revise my stance. I think the only change, for now, is the NN should only check permissions of the parent directory for deletes. The permissions of the file/dir shouldn't matter. FsShell isn't recursively descending like I thought... (Explains why deletion of a dir with millions of files took down a NN) While it's technically wrong and violates posix to allow deleting a non-empty directory to which the user has no permissions, it doesn't make anything worse. The NN's recursive delete is already terribly broken in that it ignores permissions deeper in the tree. I bet for the described problem, you could delete the no-perms dir if you created a new directory, moved the no-perms dir into it, and then recursively deleted the new directory. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11 > /user/userabc/foo-file > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file > Deleted /user/userabc/foo-file > {code} > Using "hdfs dfs -rmdir" command: > {code} > bash-4.1$ hadoop fs -lsr / > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:29 /user > drwxr-xr-x - hdfs
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13969806#comment-13969806 ] Daryn Sharp commented on HDFS-6165: --- Two-part comment to avoid overly lengthy response. The FsShell logic isn't correctly. You can't simply ignore non-empty dirs. Removal of the parent dir will fail too. I think the shell rm will then return 0 (success) even though it miserably failed. Checking for RemoteException is making assumptions about hdfs semantics, and searching for a substring is very fragile. I guess I should step back and explain how posix deletes work, although maybe everyone already knows this. There is no recursive delete. A userland tool has to recurse the directory structure and remove all children. Rm prompts when it cannot traverse a directory (lack of perms). However, the kernel will treat an empty directory just like a file - no permissions are required to delete. Rm is prompting because it doesn't know if the remove will work. (Sidenote: I tested on my mac and it will allow rmdir of a non-perm empty dir, so it's rm just being fussy and not trying to delete when -f is given even though it could) > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11 > /user/userabc/foo-file > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file > Deleted /user/userabc/foo-file >
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13969739#comment-13969739 ] Daryn Sharp commented on HDFS-6165: --- I lost track of this jira. I'll catch up today. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11 > /user/userabc/foo-file > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file > Deleted /user/userabc/foo-file > {code} > Using "hdfs dfs -rmdir" command: > {code} > bash-4.1$ hadoop fs -lsr / > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:29 /user > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:28 /user/hdfs > drwxr-xr-x - usrabc users 0 2014-03-28 23:39 /user/usrabc > drwxr-xr-x - abcabc 0 2014-03-28 23:39 > /user/usrabc/foo-empty1 > [root@vm01 usrabc]# su usrabc > [usrabc@vm01 ~]$ hdfs dfs -rmdir /user/usrabc/foo-empty1 > rmdir: Permission denied: user=usrabc, access=ALL, > inode="/user/usrabc/foo-empty1":abc:abc:drwxr-xr-x > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13969735#comment-13969735 ] Yongjun Zhang commented on HDFS-6165: - Hi Andrew, Thanks a lot for your detailed review, I will try to address them as soon as possible. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11 > /user/userabc/foo-file > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file > Deleted /user/userabc/foo-file > {code} > Using "hdfs dfs -rmdir" command: > {code} > bash-4.1$ hadoop fs -lsr / > lsr: DEPRECATED: Please use 'ls -R' instead. > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:29 /user > drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:28 /user/hdfs > drwxr-xr-x - usrabc users 0 2014-03-28 23:39 /user/usrabc > drwxr-xr-x - abcabc 0 2014-03-28 23:39 > /user/usrabc/foo-empty1 > [root@vm01 usrabc]# su usrabc > [usrabc@vm01 ~]$ hdfs dfs -rmdir /user/usrabc/foo-empty1 > rmdir: Permission denied: user=usrabc, access=ALL, > inode="/user/usrabc/foo-empty1":abc:abc:drwxr-xr-x > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13969133#comment-13969133 ] Andrew Wang commented on HDFS-6165: --- Hi Yongjun, sorry about the delay in reviewing this. Thanks for the work thus far. I think the semantics for a recursive delete via DistributedFileSystem#delete are still not quite right. The change you made will work for the shell since it does its own recursion, but we need to do the same "remove if empty dir with read" when recursing via {{recursive = true}} too. You might be able to do this by modifying {{FSPermissionChecker#checkSubAccess}} appropriately, but a new flag or new code would be safer. More comments: * isDirectory, can we add per-parameter javadoc rather than stacking on the {{@return}}? I think renaming {{empty}} to {{isEmpty}} would also help. * Nit, also need a space in the ternary {{empty?}} and {{dir.isEmptyDirectory(src)?}}. * In Delete, I think it's a bit cleaner to do an {{instanceof PathIsNotEmptyDirectoryException.class}} check instead. * Some lines longer than 80 chars TestFsShellPrivilege: I gave this a quick pass, but overall it may be better to rewrite these to use the DFS API instead of the shell. We need to test recursive delete, which the shell doesn't do, and we don't really have any shell changes in the latest rev, which lessens the importance of having new shell tests. * execCmd needs to do some try/finally to close and restore the streams if there's an exception. Also an extra commented line there. * Could we rename this file to "TestFsShellPermission"? Permission is a more standard term. * This file also should not be in hadoop-tools, but rather hadoop-common. * This does a lot of starting and stopping of a MiniCluster for running single-line tests. Can we combine these into a single test? We also don't need any DNs for this cluster, since we're just testing perms. * We have FileSystemTestHelper#createFile for creating files, can save some code * Use of @Before and @After blocks might also clarify what's going on. * This also should be a JUnit4 test with @Test annotations, not JUnit3. * USER_UGI should not be all caps, it's not static final * It's a bit ugly how we pass UNEXPECTED_RESULT in for a lot of tests. Can we just pass a boolean for {{expectSuccess}} or {{expectFailure}}, or maybe a String that we can call {{assertExceptionContains}} on? * FileEntry looks basically like a FileStatus, can we just use that instead? > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs d
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13959437#comment-13959437 ] Hadoop QA commented on HDFS-6165: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12638556/HDFS-6165.004.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-extras. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6579//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6579//console This message is automatically generated. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch, HDFS-6165.004.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > -rw-r--r-- 1 hdfsusers
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13959128#comment-13959128 ] Hadoop QA commented on HDFS-6165: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12638514/HDFS-6165.004.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-extras: org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6578//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6578//console This message is automatically generated. > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super user can delete the directory. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo > Deleted /user/userabc/foo > {code} > The same is not true for files, however. They have the correct behavior. > {code} > [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file > [root@vm01 ~]# hdfs dfs -ls /user/userabc/ > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958981#comment-13958981 ] Yongjun Zhang commented on HDFS-6165: - Hi [~daryn], Thanks for your earlier comment at 29/Mar/14 12:26. I've uploaded patch revision 004 to address them. Would you please help reviewing this version? Thanks a lot. Some notable changes and comments: 1. No -f is introduced as in previous patch revisions. 2. When deleting empty directory, in addition to the other permissions, READ permission is required for the empty directory. 3. Prior to my change, Rmdir command uses listStatus method to check whether it's empty. Curently the listStatus method requires both READ and EXECUTE permission. My patch changed the way how to check whether a directory is empty by using PathIsNotEmptyDirectoryException. 4. As stated in 3, listStatus method requires READ and EXECUTE permission. For the purpose of checking whether a dir is empty, we probably can have another lightweight API that only requires READ permission. I did not introduce this API, but I can do it if we can agree upon. My understanding is, listStatus is similar to command "ls -l". I think the new API would be similar to "ls". Basically the new API should be able to tell how many entries in a directory without the need of EXECUTE permission, below is how centOs system works, when the target directory has no EXECUTE permission: {code} [root@yjzsn usrxyz]# ls -lrt total 24 drwx-- 2 abc abc 4096 Apr 2 17:39 abc-notempty drwxr--r-- 2 abc abc 4096 Apr 2 17:39 abc-notempty1 drwx-- 2 abc abc 4096 Apr 2 17:39 abc-empty drwxr- 2 abc abc 4096 Apr 2 17:39 abc-empty1 drwx---r-- 2 abc abc 4096 Apr 2 17:39 abc-empty2 drwxr--r-- 2 abc abc 4096 Apr 2 17:39 abc-empty3 [root@yjzsn usrxyz]# su usrxyz [usrxyz@yjzsn ~]$ ls abc-notempty ls: cannot open directory abc-notempty: Permission denied [usrxyz@yjzsn ~]$ ls abc-notempty1 <== with READ perm, this one lists the file name in it (xyz.txt), prints "Permission denied" because of xyz.txt's permission ls: cannot access abc-notempty1/xyz.txt: Permission denied xyz.txt [usrxyz@yjzsn ~]$ [usrxyz@yjzsn ~]$ ls abc-empty1 ls: cannot open directory abc-empty1: Permission denied [usrxyz@yjzsn ~]$ ls abc-empty2 [usrxyz@yjzsn ~]$ ls abc-empty3 [usrxyz@yjzsn ~]$ {code} > "hdfs dfs -rm -r" and "hdfs -rmdir" commands can't remove empty directory > -- > > Key: HDFS-6165 > URL: https://issues.apache.org/jira/browse/HDFS-6165 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Minor > Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch, > HDFS-6165.003.patch, HDFS-6165.004.patch > > > Given a directory owned by user A with WRITE permission containing an empty > directory owned by user B, it is not possible to delete user B's empty > directory with either "hdfs dfs -rm -r" or "hdfs dfs -rmdir". Because the > current implementation requires FULL permission of the empty directory, and > throws exception. > On the other hand, on linux, "rm -r" and "rmdir" command can remove empty > directory as long as the parent directory has WRITE permission (and prefix > component of the path have EXECUTE permission), For the tested OSes, some > prompt user asking for confirmation, some don't. > Here's a reproduction: > {code} > [root@vm01 ~]# hdfs dfs -ls /user/ > Found 4 items > drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc > drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs > drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history > drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive > [root@vm01 ~]# hdfs dfs -ls /user/userabc > Found 8 items > drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash > drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm > drwx-- - userabc users 0 2013-05-03 01:06 > /user/userabc/.staging > drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps > drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds > drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo > drwxr-xr-x - userabc users 0 2013-04-30 16:18 > /user/userabc/maven_source > drwxr-xr-x - hdfsusers 0 2013-05-03 01:40 > /user/userabc/test-restore > [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/ > [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo > rm: Permission denied: user=userabc, access=ALL, > inode="/user/userabc/foo":hdfs:users:drwxr-xr-x > {code} > The super u