[jira] [Commented] (HDFS-5683) Better audit log messages for caching operations
[
https://issues.apache.org/jira/browse/HDFS-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954562#comment-13954562
]
Hadoop QA commented on HDFS-5683:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12637680/HDFS-5683.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6556//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6556//console
This message is automatically generated.
> Better audit log messages for caching operations
>
>
> Key: HDFS-5683
> URL: https://issues.apache.org/jira/browse/HDFS-5683
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Andrew Wang
> Labels: caching
> Attachments: HDFS-5683.001.patch
>
>
> Right now the caching audit logs aren't that useful, e.g.
> {noformat}
> 2013-12-18 14:14:54,423 INFO FSNamesystem.audit
> (FSNamesystem.java:logAuditMessage(7362)) - allowed=true ugi=andrew
> (auth:SIMPLE)ip=/127.0.0.1 cmd=addCacheDirective src=null
> dst=nullperm=null
> {noformat}
> It'd be good to include some more information when possible, like the path,
> pool, id, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" is slightly different from the Unix "rm -r" for deleting an empty directory
[
https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954555#comment-13954555
]
Hadoop QA commented on HDFS-6165:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12637675/HDFS-6165.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-tools/hadoop-extras:
org.apache.hadoop.fs.TestFilterFileSystem
org.apache.hadoop.fs.TestFilterFs
org.apache.hadoop.fs.TestHarFileSystem
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6555//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6555//console
This message is automatically generated.
> "hdfs dfs -rm -r" is slightly different from the Unix "rm -r" for deleting an
> empty directory
> -
>
> Key: HDFS-6165
> URL: https://issues.apache.org/jira/browse/HDFS-6165
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Minor
> Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch,
> HDFS-6165.003.patch
>
>
> Given a directory owned by user A with permissions 0700 containing an empty
> directory owned by user B, it is not possible to delete user B's directory.
> This is incorrect. Write permission on the containing directory should be all
> that is needed to delete the child directory. Here's a reproduction:
> {code}
> [root@vm01 ~]# hdfs dfs -ls /user/
> Found 4 items
> drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc
> drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs
> drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history
> drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive
> [root@vm01 ~]# hdfs dfs -ls /user/userabc
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users 0 2013-05-03 01:06
> /user/userabc/.staging
> drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
> drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo
> drwxr-xr-x - userabc users 0 2013-04-30 16:18
> /user/userabc/maven_source
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:40
> /user/userabc/test-restore
> [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/
> [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo
> rm: Permission denied: user=userabc, access=ALL,
> inode="/user/userabc/foo":hdfs:users:drwxr-xr-x
> {code}
> The super user can delete the directory.
> {code}
> [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo
> Deleted /user/userabc/foo
> {code}
> The same is not true for files, however. They have the correct behavior.
> {code}
> [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file
> [root@vm01 ~]# hdfs dfs -ls /user/userabc/
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users 0 2013-05-03 01:06
> /user/userabc/.staging
> drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
> drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds
> -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11
> /user/userabc/foo-file
> drwxr-xr-x - userabc users 0 2013-04-30 16:18
> /user/userabc/maven_source
> drwxr-xr-x - hdf
[jira] [Updated] (HDFS-5683) Better audit log messages for caching operations
[
https://issues.apache.org/jira/browse/HDFS-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhiraj Butala updated HDFS-5683:
-
Status: Patch Available (was: Open)
> Better audit log messages for caching operations
>
>
> Key: HDFS-5683
> URL: https://issues.apache.org/jira/browse/HDFS-5683
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.3.0, 3.0.0
>Reporter: Andrew Wang
> Labels: caching
> Attachments: HDFS-5683.001.patch
>
>
> Right now the caching audit logs aren't that useful, e.g.
> {noformat}
> 2013-12-18 14:14:54,423 INFO FSNamesystem.audit
> (FSNamesystem.java:logAuditMessage(7362)) - allowed=true ugi=andrew
> (auth:SIMPLE)ip=/127.0.0.1 cmd=addCacheDirective src=null
> dst=nullperm=null
> {noformat}
> It'd be good to include some more information when possible, like the path,
> pool, id, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-5683) Better audit log messages for caching operations
[
https://issues.apache.org/jira/browse/HDFS-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhiraj Butala updated HDFS-5683:
-
Attachment: HDFS-5683.001.patch
I addressed the above issue and made appropriate changes to FSNamesystem.java.
Giving below some examples of the caching audit logs with the patch:
{noformat}
14/03/29 02:00:59 INFO FSNamesystem.audit: allowed=true ugi=abutala
(auth:SIMPLE) ip=/127.0.0.1 cmd=addCacheDirective src={id: 8, path:
/user/abutala/abhiraj, replication: 1, pool: pool7, expiration:
73071270-05-24T21:49:13-0700} dst=nullperm=null
14/03/29 02:01:49 INFO FSNamesystem.audit: allowed=true ugi=abutala
(auth:SIMPLE) ip=/127.0.0.1 cmd=modifyCacheDirectivesrc={id: 8}
dst={id: 8, path: /user/abutala/abhiraj/tmp2} perm=null
14/03/29 02:03:35 INFO FSNamesystem.audit: allowed=true ugi=abutala
(auth:SIMPLE) ip=/127.0.0.1 cmd=listCacheDirectives src={} dst=null
perm=null
14/03/29 02:03:47 INFO FSNamesystem.audit: allowed=true ugi=abutala
(auth:SIMPLE) ip=/127.0.0.1 cmd=listCacheDirectives src={pool: pool2}
dst=nullperm=null
14/03/29 02:04:02 INFO FSNamesystem.audit: allowed=true ugi=abutala
(auth:SIMPLE) ip=/127.0.0.1 cmd=listCacheDirectives src={path:
/user/abutala/abhiraj, pool: pool2} dst=nullperm=null
14/03/29 02:05:54 INFO FSNamesystem.audit: allowed=true ugi=abutala
(auth:SIMPLE) ip=/127.0.0.1 cmd=removeCacheDirectivesrc={id: 8}
dst=nullperm=null
14/03/29 02:08:44 INFO FSNamesystem.audit: allowed=true ugi=abutala
(auth:SIMPLE) ip=/127.0.0.1 cmd=addCachePool
src={poolName:pool10, ownerName:abutala, groupName:abutala, mode:0755,
limit:9223372036854775807, maxRelativeExpiryMs:2305843009213693951}
dst=nullperm=null
14/03/29 02:09:58 INFO FSNamesystem.audit: allowed=true ugi=abutala
(auth:SIMPLE) ip=/127.0.0.1 cmd=modifyCachePool src={poolName:
pool10} dst={poolName:pool10, ownerName:null, groupName:null, mode:0666,
limit:null, maxRelativeExpiryMs:null} perm=null
14/03/29 02:11:21 INFO FSNamesystem.audit: allowed=true ugi=abutala
(auth:SIMPLE) ip=/127.0.0.1 cmd=removeCachePool src={poolName:
pool10} dst=nullperm=null
{noformat}
For modifyCacheDirective and modifyCachePool, I put the final changes in 'dst'
section and the 'src' section only has the id or pool name being modified
respectively. Also, not including any tests as this is just an update to the
logs.
Kindly review and let me know if there are any issues. Thank you!
> Better audit log messages for caching operations
>
>
> Key: HDFS-5683
> URL: https://issues.apache.org/jira/browse/HDFS-5683
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Andrew Wang
> Labels: caching
> Attachments: HDFS-5683.001.patch
>
>
> Right now the caching audit logs aren't that useful, e.g.
> {noformat}
> 2013-12-18 14:14:54,423 INFO FSNamesystem.audit
> (FSNamesystem.java:logAuditMessage(7362)) - allowed=true ugi=andrew
> (auth:SIMPLE)ip=/127.0.0.1 cmd=addCacheDirective src=null
> dst=nullperm=null
> {noformat}
> It'd be good to include some more information when possible, like the path,
> pool, id, etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6165) "hdfs dfs -rm -r" is slightly different from the Unix "rm -r" for deleting an empty directory
[
https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-6165:
Attachment: HDFS-6165.003.patch
While we are still discussing about the solution, I'm uploading a new revision
003 to resolve the missed stuff from previous one.
> "hdfs dfs -rm -r" is slightly different from the Unix "rm -r" for deleting an
> empty directory
> -
>
> Key: HDFS-6165
> URL: https://issues.apache.org/jira/browse/HDFS-6165
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Minor
> Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch,
> HDFS-6165.003.patch
>
>
> Given a directory owned by user A with permissions 0700 containing an empty
> directory owned by user B, it is not possible to delete user B's directory.
> This is incorrect. Write permission on the containing directory should be all
> that is needed to delete the child directory. Here's a reproduction:
> {code}
> [root@vm01 ~]# hdfs dfs -ls /user/
> Found 4 items
> drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc
> drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs
> drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history
> drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive
> [root@vm01 ~]# hdfs dfs -ls /user/userabc
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users 0 2013-05-03 01:06
> /user/userabc/.staging
> drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
> drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo
> drwxr-xr-x - userabc users 0 2013-04-30 16:18
> /user/userabc/maven_source
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:40
> /user/userabc/test-restore
> [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/
> [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo
> rm: Permission denied: user=userabc, access=ALL,
> inode="/user/userabc/foo":hdfs:users:drwxr-xr-x
> {code}
> The super user can delete the directory.
> {code}
> [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo
> Deleted /user/userabc/foo
> {code}
> The same is not true for files, however. They have the correct behavior.
> {code}
> [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file
> [root@vm01 ~]# hdfs dfs -ls /user/userabc/
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users 0 2013-05-03 01:06
> /user/userabc/.staging
> drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
> drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds
> -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11
> /user/userabc/foo-file
> drwxr-xr-x - userabc users 0 2013-04-30 16:18
> /user/userabc/maven_source
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:40
> /user/userabc/test-restore
> [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file
> Deleted /user/userabc/foo-file
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[
https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954475#comment-13954475
]
Hadoop QA commented on HDFS-4564:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635621/HDFS-4564.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6554//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6554//console
This message is automatically generated.
> Webhdfs returns incorrect http response codes for denied operations
> ---
>
> Key: HDFS-4564
> URL: https://issues.apache.org/jira/browse/HDFS-4564
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: webhdfs
>Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Blocker
> Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch,
> HDFS-4564.branch-23.patch, HDFS-4564.patch, HDFS-4564.patch
>
>
> Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's
> denying operations. Examples including rejecting invalid proxy user attempts
> and renew/cancel with an invalid user.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" is slightly different from the Unix "rm -r" for deleting an empty directory
[
https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954431#comment-13954431
]
Yongjun Zhang commented on HDFS-6165:
-
HI [~daryn],
Thanks a lot for your experiments and comments. The read permission thing is
indeed what's not noticed earlier. It definitely need to be addressed. Several
follow-up questions here:
1. My understanding of your point is, if the empty sub-dir has read permission,
then "-rm " and "-rmdir" would simply go ahead remove the dir, even
though all linux systems we tested so far prompt user (when possible), right?
If the answer is yes, then the userland won't have a choice whether it should
prompt to user (as in your summary point #2). I think this solution works fine
for me, but would you please confirm? The folks who commented earlier may
comment on this too.
2. Not to say that we have to do this, for my better understanding, does adding
-f support to the NN protocol as an optional field break compatibility? (my bad
in version 002, I did miss some places to change, the scope seems to be wide
because it's an additional API that I'm adding).
3. Just to clarify, I actually tried to avoid prompting with the patch. At
occasions when linux prompts, the patch I did throws exception like before,
however, with improved message to suggestion user to try -f switch.
BTW, your second to last experiment "Read perms sans -f require a prompt??"
didn't really use "-f" switch, but sounds like you intended to use "-f" in
this test, if so, would you please retry this one with "-f"?
Thanks again.
> "hdfs dfs -rm -r" is slightly different from the Unix "rm -r" for deleting an
> empty directory
> -
>
> Key: HDFS-6165
> URL: https://issues.apache.org/jira/browse/HDFS-6165
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Minor
> Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch
>
>
> Given a directory owned by user A with permissions 0700 containing an empty
> directory owned by user B, it is not possible to delete user B's directory.
> This is incorrect. Write permission on the containing directory should be all
> that is needed to delete the child directory. Here's a reproduction:
> {code}
> [root@vm01 ~]# hdfs dfs -ls /user/
> Found 4 items
> drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc
> drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs
> drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history
> drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive
> [root@vm01 ~]# hdfs dfs -ls /user/userabc
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users 0 2013-05-03 01:06
> /user/userabc/.staging
> drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
> drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo
> drwxr-xr-x - userabc users 0 2013-04-30 16:18
> /user/userabc/maven_source
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:40
> /user/userabc/test-restore
> [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/
> [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo
> rm: Permission denied: user=userabc, access=ALL,
> inode="/user/userabc/foo":hdfs:users:drwxr-xr-x
> {code}
> The super user can delete the directory.
> {code}
> [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo
> Deleted /user/userabc/foo
> {code}
> The same is not true for files, however. They have the correct behavior.
> {code}
> [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file
> [root@vm01 ~]# hdfs dfs -ls /user/userabc/
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users 0 2013-05-03 01:06
> /user/userabc/.staging
> drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
> drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds
> -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11
> /user/userabc/foo-file
> drwxr-xr-x - userabc users 0 2013-04-30 16:18
> /user/userabc/maven_source
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:40
> /user/userabc/test-restore
> [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTr
[jira] [Commented] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[ https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954421#comment-13954421 ] Jing Zhao commented on HDFS-4564: - Thanks for the clarification, [~daryn]. The current patch looks good to me, +1. We may need more cleanup and fix with AuthenticatedURL related code, but I think we can do it in separate jiras. So have you tested the patch internally? Could you please generally mention how you tested the patch? > Webhdfs returns incorrect http response codes for denied operations > --- > > Key: HDFS-4564 > URL: https://issues.apache.org/jira/browse/HDFS-4564 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch, > HDFS-4564.branch-23.patch, HDFS-4564.patch, HDFS-4564.patch > > > Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's > denying operations. Examples including rejecting invalid proxy user attempts > and renew/cancel with an invalid user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[
https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954419#comment-13954419
]
Haohui Mai commented on HDFS-4564:
--
bq. It does, but it creates an incredible noisy exception trace that reports
the failure as the response not being json.
It does not seem to me to be a blocker of 2.4. Can you please move it out to
2.5?
bq. Bottom line is AuthenticatedURL is unnecessary and introduces nothing but
problems for webhdfs. It's only useful for oozie's anon/non-anon support.
{{URLConnectionFactory}} wraps the same mechanism, and it is used implicitly in
transferring edit logs between NN and JNs, transferring fsimage between NNs,
etc. If you think that this is a no-op, it might be better to revisit
{{URLConnectionFactory}} and to clean up all the issues in one jira. Addressing
it in webhdfs only looks a partial solution to me.
In my own opinion, I don't think the above issues are blockers. Maybe we can
move forward and address them in 2.5.
> Webhdfs returns incorrect http response codes for denied operations
> ---
>
> Key: HDFS-4564
> URL: https://issues.apache.org/jira/browse/HDFS-4564
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: webhdfs
>Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Blocker
> Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch,
> HDFS-4564.branch-23.patch, HDFS-4564.patch, HDFS-4564.patch
>
>
> Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's
> denying operations. Examples including rejecting invalid proxy user attempts
> and renew/cancel with an invalid user.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" is slightly different from the Unix "rm -r" for deleting an empty directory
[
https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954405#comment-13954405
]
Daryn Sharp commented on HDFS-6165:
---
Here's what I get on my mac (BSD based 10.9) which seems similar to the
previously posted results. It's a full recap to spare others reading this
whole discussion.
Fails because no perms at all to see dir contents.
{noformat}
$ mkdir -p test/bar; sudo chown root test/bar; sudo chmod a= test/bar
$ rm -rf test
rm: test/bar: Permission denied
rm: test: Directory not empty
{noformat}
Fails because execute but no read perms to see dir contents.
{noformat}
$ mkdir -p test/bar; sudo chown root test/bar; sudo chmod a=x test/bar
$ rm -rf test
rm: test/bar: Permission denied
rm: test: Directory not empty
{noformat}
Works because of read perms and -f flag.
{noformat}
$ mkdir -p test/bar; sudo chown root test/bar; sudo chmod a=r test/bar
$ rm -rf test
{noformat}
Read perms sans -f require a prompt??
{noformat}
$ mkdir -p test/bar; sudo chown root test/bar; sudo chmod a=r test/bar
$ rm -r test
override r--r--r-- root/wheel for test/bar?
{noformat}
Works because can't prompt with no STDIN.
{noformat}
$ mkdir -p test/bar; sudo chown root test/bar; sudo chmod a=r test/bar
$ perl -e 'close(STDIN); system("rm -r test")'
{noformat}
---
Quick summary:
# the kernel will delete a non-owned directory if and only if the user has read
perms which allow it to determine the directory is empty
# it's the userland rm that decides if it should prompt to delete a non-owned
directory
I don't agree with any incompatible changes to the NN protocol. Years ago I
made FsShell as POSIX like as possible and have always argued for POSIX
behavior since. However, I find the prompting silly and incompatible. It's a
userland check so adding this support to the NN doesn't make sense to me.
My opinion is the only change should be the NN allows empty dirs be deleted if
the user has read perms. No changes to -f. Let -f continue to mean "If the
file does not exist, do not display a diagnostic message or modify the exit
status to reflect an error".
> "hdfs dfs -rm -r" is slightly different from the Unix "rm -r" for deleting an
> empty directory
> -
>
> Key: HDFS-6165
> URL: https://issues.apache.org/jira/browse/HDFS-6165
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Minor
> Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch
>
>
> Given a directory owned by user A with permissions 0700 containing an empty
> directory owned by user B, it is not possible to delete user B's directory.
> This is incorrect. Write permission on the containing directory should be all
> that is needed to delete the child directory. Here's a reproduction:
> {code}
> [root@vm01 ~]# hdfs dfs -ls /user/
> Found 4 items
> drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc
> drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs
> drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history
> drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive
> [root@vm01 ~]# hdfs dfs -ls /user/userabc
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users 0 2013-05-03 01:06
> /user/userabc/.staging
> drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
> drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo
> drwxr-xr-x - userabc users 0 2013-04-30 16:18
> /user/userabc/maven_source
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:40
> /user/userabc/test-restore
> [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/
> [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo
> rm: Permission denied: user=userabc, access=ALL,
> inode="/user/userabc/foo":hdfs:users:drwxr-xr-x
> {code}
> The super user can delete the directory.
> {code}
> [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo
> Deleted /user/userabc/foo
> {code}
> The same is not true for files, however. They have the correct behavior.
> {code}
> [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file
> [root@vm01 ~]# hdfs dfs -ls /user/userabc/
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users
[jira] [Commented] (HDFS-3503) Move LengthInputStream and PositionTrackingInputStream to common
[
https://issues.apache.org/jira/browse/HDFS-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954394#comment-13954394
]
Hadoop QA commented on HDFS-3503:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12637591/h3503_20140328.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6552//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6552//console
This message is automatically generated.
> Move LengthInputStream and PositionTrackingInputStream to common
>
>
> Key: HDFS-3503
> URL: https://issues.apache.org/jira/browse/HDFS-3503
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Attachments: h3503_20140328.patch
>
>
> We have LengthInputStream in org.apache.hadoop.hdfs.server.datanode.fsdataset
> and PositionTrackingInputStream in
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader. These two classes
> are generally useful. Let's move them to org.apache.hadoop.io.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[ https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954390#comment-13954390 ] Daryn Sharp commented on HDFS-4564: --- bq. Correct me if I'm wrong, but it looks like the following code handles the case you mentioned. It does, but it creates an incredible noisy exception trace that reports the failure as the response not being json. The specific throw provides a clean "authenticated required" message. bq. The current code also covers this for explicitly getting, renewing, or canceling a token. Your patch changes this part and pass false to the url factory so it bypasses the use of authenticated url (per your comments in HADOOP-10301). Could you give more details how it works without using the authenticated url? The current code only checks the TGT implicitly via AuthenticatedURL. Prior to this patch, AuthenticatedURL is only used for token operations. All other operations do not use it since they authenticate with a token. With this patch, AuthenticatedURL is not used because it is buggy in part to causing replay attacks, double attempts to kerberos authenticate with the fallback authenticator if the TGT is expired, incorrectly uses the fallback authenticator (required by oozie servers) to add the username parameter which webhdfs has already included in the uri. AuthenticatedURL's attempt to do SPNEGO auth is a no-op because the JDK transparently does SPNEGO when the user's Subject (UGI) contains kerberos principals. Since AuthenticatedURL is now not used, webhdfs has to check the TGT itself for token operations. Bottom line is AuthenticatedURL is unnecessary and introduces nothing but problems for webhdfs. It's only useful for oozie's anon/non-anon support. > Webhdfs returns incorrect http response codes for denied operations > --- > > Key: HDFS-4564 > URL: https://issues.apache.org/jira/browse/HDFS-4564 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch, > HDFS-4564.branch-23.patch, HDFS-4564.patch, HDFS-4564.patch > > > Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's > denying operations. Examples including rejecting invalid proxy user attempts > and renew/cancel with an invalid user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[ https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954374#comment-13954374 ] Arun C Murthy commented on HDFS-4564: - [~daryn] - Can you pls clarify some of the above review comments? I'd like to move fwd with 2.4 ASAP. Tx! > Webhdfs returns incorrect http response codes for denied operations > --- > > Key: HDFS-4564 > URL: https://issues.apache.org/jira/browse/HDFS-4564 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch, > HDFS-4564.branch-23.patch, HDFS-4564.patch, HDFS-4564.patch > > > Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's > denying operations. Examples including rejecting invalid proxy user attempts > and renew/cancel with an invalid user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6165) "hdfs dfs -rm -r" is slightly different from the Unix "rm -r" for deleting an empty directory
[
https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954372#comment-13954372
]
Hadoop QA commented on HDFS-6165:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12637658/HDFS-6165.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6553//console
This message is automatically generated.
> "hdfs dfs -rm -r" is slightly different from the Unix "rm -r" for deleting an
> empty directory
> -
>
> Key: HDFS-6165
> URL: https://issues.apache.org/jira/browse/HDFS-6165
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Minor
> Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch
>
>
> Given a directory owned by user A with permissions 0700 containing an empty
> directory owned by user B, it is not possible to delete user B's directory.
> This is incorrect. Write permission on the containing directory should be all
> that is needed to delete the child directory. Here's a reproduction:
> {code}
> [root@vm01 ~]# hdfs dfs -ls /user/
> Found 4 items
> drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc
> drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs
> drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history
> drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive
> [root@vm01 ~]# hdfs dfs -ls /user/userabc
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users 0 2013-05-03 01:06
> /user/userabc/.staging
> drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
> drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:54 /user/userabc/foo
> drwxr-xr-x - userabc users 0 2013-04-30 16:18
> /user/userabc/maven_source
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:40
> /user/userabc/test-restore
> [root@vm01 ~]# hdfs dfs -ls /user/userabc/foo/
> [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -r -skipTrash /user/userabc/foo
> rm: Permission denied: user=userabc, access=ALL,
> inode="/user/userabc/foo":hdfs:users:drwxr-xr-x
> {code}
> The super user can delete the directory.
> {code}
> [root@vm01 ~]# sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/userabc/foo
> Deleted /user/userabc/foo
> {code}
> The same is not true for files, however. They have the correct behavior.
> {code}
> [root@vm01 ~]# sudo -u hdfs hdfs dfs -touchz /user/userabc/foo-file
> [root@vm01 ~]# hdfs dfs -ls /user/userabc/
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users 0 2013-05-03 01:06
> /user/userabc/.staging
> drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
> drwxr-xr-x - userabc users 0 2013-04-30 18:05 /user/userabc/ds
> -rw-r--r-- 1 hdfsusers 0 2013-05-03 02:11
> /user/userabc/foo-file
> drwxr-xr-x - userabc users 0 2013-04-30 16:18
> /user/userabc/maven_source
> drwxr-xr-x - hdfsusers 0 2013-05-03 01:40
> /user/userabc/test-restore
> [root@vm01 ~]# sudo -u userabc hdfs dfs -rm -skipTrash /user/userabc/foo-file
> Deleted /user/userabc/foo-file
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6165) "hdfs dfs -rm -r" is slightly different from the Unix "rm -r" for deleting an empty directory
[
https://issues.apache.org/jira/browse/HDFS-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-6165:
Attachment: HDFS-6165.002.patch
Thank you all for the comments and discussions and reviews.
Uploaded version 002 to fix all three issues summarized in my last update.
There are a few questions to ask you experienced folks:
1. This fix involves changing ClientNamenodeProtocol.proto by adding an
optional bit to message DeleteRequestProto to indicate whether "-f" is
specified. Is there any implication of API compatibility here?
2. This change involves both files in hadoop-common-project and
hadoop-hdfs-project, is it fine to be covered by this single JIRA?
3. To address Andrew's comments on how the test is written, I discussed with
him, and we had an agreement that it would be beneficial to have this new
testing infrastructure for FsShelll operations (TestFsShellPrivilege.java
introduced with this fix). It has the advantage of manipulating file structure,
ownership, permission, and calling FsShell with command line arguments, and
easy debugging with eclipse.
My test result with the fix:
{code}
[usrxyz@vm001 hadoop]$ hadoop fs -lsr /
lsr: DEPRECATED: Please use 'ls -R' instead.
drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:29 /user
drwxr-xr-x - hdfs supergroup 0 2014-03-25 16:28 /user/hdfs
drwxr-xr-x - usrxyz users 0 2014-03-29 10:19 /user/usrxyz
drwxr-xr-x - abcusers 0 2014-03-25 16:32 /user/usrxyz/foo
-rw-r--r-- 1 abcusers 5 2014-03-25 16:32
/user/usrxyz/foo/test.txt
drwxr-xr-x - abcusers 0 2014-03-29 10:18
/user/usrxyz/foo-empty
drwxr-xr-x - abcusers 0 2014-03-29 10:18
/user/usrxyz/foo-empty1
drwxr-xr-x - abcusers 0 2014-03-29 10:19
/user/usrxyz/foo-empty2
[usrxyz@vm001 hadoop]$ whoami
usrxyz
[usrxyz@vm001 hadoop]$ hdfs dfs -rm -r /user/usrxyz/foo
14/03/29 10:20:42 INFO fs.TrashPolicyDefault: Namenode trash configuration:
Deletion interval = 0 minutes, Emptier interval = 0 minutes.
rm: Permission denied: user=usrxyz, access=ALL,
inode="/user/usrxyz/foo":abc:users:drwxr-xr-x
[usrxyz@vm001 hadoop]$ hdfs dfs -rm -r /user/usrxyz/foo-empty
14/03/29 10:20:54 INFO fs.TrashPolicyDefault: Namenode trash configuration:
Deletion interval = 0 minutes, Emptier interval = 0 minutes.
rm: Permission denied: user=usrxyz, access=ALL,
inode="/user/usrxyz/foo-empty":abc:users:drwxr-xr-x. Consider trying again with
-f switch.
[usrxyz@vm001 hadoop]$ hdfs dfs -rm -r -f /user/usrxyz/foo-empty
14/03/29 10:21:06 INFO fs.TrashPolicyDefault: Namenode trash configuration:
Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /user/usrxyz/foo-empty
[usrxyz@vm001 hadoop]$ hdfs dfs -rmdir /user/usrxyz/foo-empty1
[usrxyz@vm001 hadoop]$ hdfs dfs -rmdir /user/usrxyz/foo
rmdir: `/user/usrxyz/foo is non empty
[usrxyz@vm001 hadoop]$ whoami
usrxyz
[usrxyz@vm001 hadoop]$ su abc
Password:
[abc@vm001 hadoop]$ whoami
abc
[abc@vm001 hadoop]$ hdfs dfs -rmdir /user/usrxyz/foo-empty2
rmdir: Permission denied: user=abc, access=WRITE,
inode="/user/usrxyz":usrxyz:users:drwxr-xr-x
{code}
Thanks a lot for reviewing the fix.
> "hdfs dfs -rm -r" is slightly different from the Unix "rm -r" for deleting an
> empty directory
> -
>
> Key: HDFS-6165
> URL: https://issues.apache.org/jira/browse/HDFS-6165
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.3.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Minor
> Attachments: HDFS-6165.001.patch, HDFS-6165.002.patch
>
>
> Given a directory owned by user A with permissions 0700 containing an empty
> directory owned by user B, it is not possible to delete user B's directory.
> This is incorrect. Write permission on the containing directory should be all
> that is needed to delete the child directory. Here's a reproduction:
> {code}
> [root@vm01 ~]# hdfs dfs -ls /user/
> Found 4 items
> drwxr-xr-x - userabc users 0 2013-05-03 01:55 /user/userabc
> drwxr-xr-x - hdfssupergroup 0 2013-05-03 00:28 /user/hdfs
> drwxrwxrwx - mapred hadoop 0 2013-05-03 00:13 /user/history
> drwxr-xr-x - hdfssupergroup 0 2013-04-14 16:46 /user/hive
> [root@vm01 ~]# hdfs dfs -ls /user/userabc
> Found 8 items
> drwx-- - userabc users 0 2013-05-02 17:00 /user/userabc/.Trash
> drwxr-xr-x - userabc users 0 2013-05-03 01:34 /user/userabc/.cm
> drwx-- - userabc users 0 2013-05-03 01:06
> /user/userabc/.staging
> drwxr-xr-x - userabc users 0 2013-04-14 18:31 /user/userabc/apps
> drwxr-xr-x - userabc users
[jira] [Commented] (HDFS-6166) revisit balancer so_timeout
[ https://issues.apache.org/jira/browse/HDFS-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954326#comment-13954326 ] Hudson commented on HDFS-6166: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5431 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5431/]) HDFS-6166. Change Balancer socket read timeout to 20 minutes and add 10 seconds delay after error. Contributed by Nathan Roberts (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1583018) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java > revisit balancer so_timeout > > > Key: HDFS-6166 > URL: https://issues.apache.org/jira/browse/HDFS-6166 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 3.0.0, 2.3.0 >Reporter: Nathan Roberts >Assignee: Nathan Roberts >Priority: Blocker > Fix For: 2.4.0 > > Attachments: HDFS-6166.patch > > > HDFS-5806 changed the socket read timeout for the balancer connection to DN > to 60 seconds. This works as long as balancer bandwidth is such that it's > safe to assume that the DN will easily complete the operation within this > time. Obviously this isn't a good assumption. When this assumption isn't > valid, the balancer will timeout the cmd BUT it will then be out-of-sync with > the datanode (balancer thinks the DN has room to do more work, DN is still > working on the request and will fail any subsequent requests with "threads > quota exceeded errors"). This causes expensive NN traffic via getBlocks() and > also causes lots of WARNS int the balancer log. > Unfortunately the protocol is such that it's impossible to tell if the DN is > busy working on replacing the block, OR is in bad shape and will never finish. > So, in the interest of a small change to deal with both situations, I propose > the following two changes: > * Crank of the socket read timeout to 20 minutes > * Delay looking at a node for a bit if we did timeout in this way (the DN > could still have xceiver threads working on the replace -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6166) revisit balancer so_timeout
[ https://issues.apache.org/jira/browse/HDFS-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-6166: -- Resolution: Fixed Fix Version/s: 2.4.0 Status: Resolved (was: Patch Available) The failed test is not related. I have committed this. Thanks, Nathan! > revisit balancer so_timeout > > > Key: HDFS-6166 > URL: https://issues.apache.org/jira/browse/HDFS-6166 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 3.0.0, 2.3.0 >Reporter: Nathan Roberts >Assignee: Nathan Roberts >Priority: Blocker > Fix For: 2.4.0 > > Attachments: HDFS-6166.patch > > > HDFS-5806 changed the socket read timeout for the balancer connection to DN > to 60 seconds. This works as long as balancer bandwidth is such that it's > safe to assume that the DN will easily complete the operation within this > time. Obviously this isn't a good assumption. When this assumption isn't > valid, the balancer will timeout the cmd BUT it will then be out-of-sync with > the datanode (balancer thinks the DN has room to do more work, DN is still > working on the request and will fail any subsequent requests with "threads > quota exceeded errors"). This causes expensive NN traffic via getBlocks() and > also causes lots of WARNS int the balancer log. > Unfortunately the protocol is such that it's impossible to tell if the DN is > busy working on replacing the block, OR is in bad shape and will never finish. > So, in the interest of a small change to deal with both situations, I propose > the following two changes: > * Crank of the socket read timeout to 20 minutes > * Delay looking at a node for a bit if we did timeout in this way (the DN > could still have xceiver threads working on the replace -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6156) Simplify the JMX API that provides snapshot information
[
https://issues.apache.org/jira/browse/HDFS-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954253#comment-13954253
]
Hudson commented on HDFS-6156:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1741 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1741/])
HDFS-6156. Simplify the JMX API that provides snapshot information. Contributed
by Shinichi Yamashita. (wheat9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1582847)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshotInfo.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshottableDirectoryStatus.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotStatsMXBean.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotStatsMXBean.java
> Simplify the JMX API that provides snapshot information
> ---
>
> Key: HDFS-6156
> URL: https://issues.apache.org/jira/browse/HDFS-6156
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Shinichi Yamashita
> Fix For: 2.5.0
>
> Attachments: HDFS-6156-2.patch, HDFS-6156.patch
>
>
> HDFS-5196 introduces a set of new APIs that provide snapshot information
> through JMX. Currently. The API nests {{SnapshotDirectoryMXBean}} into
> {{SnapshotStatsMXBean}}, creating another layer of composition.
> This jira proposes to inline {{SnapshotDirectoryMXBean}} into
> {{SnapshotStatsMXBean}} and to simplify the API.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6168) Remove deprecated methods in DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954248#comment-13954248 ] Hudson commented on HDFS-6168: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1741 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1741/]) HDFS-6168. Remove a deprecated constructor and the deprecated methods reportChecksumFailure, getDelegationToken(Text), renewDelegationToken and cancelDelegationToken from DistributedFileSystem. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1582856) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java > Remove deprecated methods in DistributedFileSystem > -- > > Key: HDFS-6168 > URL: https://issues.apache.org/jira/browse/HDFS-6168 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.5.0 > > Attachments: h6168_20140327.patch, h6168_20140327b.patch > > > Some methods in DistributedFileSystem are already deprecated for a long time. > They should be removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6010) Make balancer able to balance data among specified servers
[
https://issues.apache.org/jira/browse/HDFS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951867#comment-13951867
]
Yu Li commented on HDFS-6010:
-
Thanks for the review and comments Tsz.
{quote}
I think "-datanodes" may be a better name than "-servers"...How about adding a
new conf property, say dfs.balancer.selectedDatanodes?
{quote}
IMHO, by making it an option in CLI, user could dynamically choose which nodes
to balance among, while property is static. In our use case, the admin might
balance groupA and groupB separately, and an option in CLI would make it
easier, right?
Agree to rename the option as "-datanodes" if we decided to still use option in
CLI.
{quote}
How about moving it to the balancer package and renaming it to BalancerUtil?
{quote}
Agree to move it to balancer package. About the name, since currently it's only
for validating whether a given string matches a live datanode, it seems to me
the name "BalancerUtil" is too big. :-)
{quote}
a balancer may run for a long time and some datanodes could be down. I think we
should not throw exceptions. Perhaps, printing a warning is good enough
{quote}
It's true tat some datanodes could be down, but I'd like to discuss more about
this scenario. Assuming groupA has 3 nodes and node #1 is down. When admin
issue command like "-datanodes 1,2,3", he means to make data distribution got
balanced across the 3 nodes. If we only print warnings, then it will balance
data between node #2 and #3 firstly, then after node #1 is back, the admin has
to do another round of balancing. Since each balance would add read lock to
involved blocks and cause disk/network IO, in our product env we would prefer
to fail the first trial and wait until all datanodes back. So I'd like to ask
for a second thought on whether to throw exception or print warning here.
{quote}
The new code could be moved to a static method (in BalancerUtil) so that it is
earlier to read.
{quote}
Agree, will refine the code no matter whether we need to change from throwing
exception to printing warning
{quote}
I have not yet checked NodeStringValidator and the new tests in details
{quote}
No problem, will wait for your comments and update the patch in one go, along
with all changes required after above discussion.
> Make balancer able to balance data among specified servers
> --
>
> Key: HDFS-6010
> URL: https://issues.apache.org/jira/browse/HDFS-6010
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer
>Affects Versions: 2.3.0
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Minor
> Labels: balancer
> Attachments: HDFS-6010-trunk.patch, HDFS-6010-trunk_V2.patch
>
>
> Currently, the balancer tool balances data among all datanodes. However, in
> some particular case, we would need to balance data only among specified
> nodes instead of the whole set.
> In this JIRA, a new "-servers" option would be introduced to implement this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
