[jira] [Commented] (HADOOP-13771) Adding group mapping lookup utility without dependency on HDFS namenode
[
https://issues.apache.org/jira/browse/HADOOP-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626413#comment-15626413
]
Allen Wittenauer commented on HADOOP-13771:
---
What I had in my head is that if 'hdfs groups' can't contact the NN after a
much shorter timeout, it would then run locally and provide an answer with the
additional text of "(non-authoritative)" or something else in the output.
> Adding group mapping lookup utility without dependency on HDFS namenode
> ---
>
> Key: HADOOP-13771
> URL: https://issues.apache.org/jira/browse/HADOOP-13771
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, tools
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-13771.00.patch
>
>
> We have {{hdfs groups}} command to troubleshoot issues related to users'
> group member look up with Unix/LDAP. However, there are some limitation of
> this command: 1) it can only be executed when namenode is running. 2) any
> change in the group mapping lookup configuration needs a hdfs namenode
> restart, which is expensive.
> This ticket is proposed to have a simple CLI utility like HadoopKerberosName
> {code}
> hadoop org.apache.hadoop.security.HadoopKerberosName
> nn/localh...@hdpdev.dev.com
> {code}
> The CLI utility for group member lookup will have a usage like below without
> namenode running or restart for configuration change.
> {code}
> hadoop org.apache.hadoop.security.Groups hdfs
> hdfs : [hadoop, hdfs]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13771) Adding group mapping lookup utility without dependency on HDFS namenode
[
https://issues.apache.org/jira/browse/HADOOP-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626392#comment-15626392
]
Xiaoyu Yao commented on HADOOP-13771:
-
Thanks [~aw] for providing the details of "hdfs groups". That well explained
why we have "hdfs groups" instead of "hadoop groups" today.
bq. There's an additional wrinkle here. The NN is not the only process that is
doing group resolution. Pretty much any service that does ACL resolution also
does group resolution to some degree. Making the command 'hadoop groups' is
going lead some folks to think that this works for any service...
Agree. Based on your description, I would prefer keep "hdfs groups" as-is today
instead of replacing "hdfs groups".
How about expose this as a DEBUG tool only? Below are some choices here:
1) run only with class Main only, no CLI exposed.
hadoop org.apache.hadoop.security.Groups
2) Add "hadoop groups" which wraps 1) in script, less ideal as you mentioned
above.
3) Add "hdfs debug groups" which wrapps 1) in script.
Explicitly mention the result is solely based on the configurations from
core-site.xml configurations.
It is authoritative compared with "hdfs groups"
bq. I'd therefore propose a different solution. 'hdfs groups' should work like
nslookup. If the NN is up, it should query the NN and give an authoritative
answer. If the NN is not up, it should give the local answer but be absolutely
clear that it is at best a guess and may be in correct.
This proposal looks good to me as well. MR and HDFS share a common base for the
"group" lookup. This will change the group lookup for both HDFS and MR.
> Adding group mapping lookup utility without dependency on HDFS namenode
> ---
>
> Key: HADOOP-13771
> URL: https://issues.apache.org/jira/browse/HADOOP-13771
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, tools
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-13771.00.patch
>
>
> We have {{hdfs groups}} command to troubleshoot issues related to users'
> group member look up with Unix/LDAP. However, there are some limitation of
> this command: 1) it can only be executed when namenode is running. 2) any
> change in the group mapping lookup configuration needs a hdfs namenode
> restart, which is expensive.
> This ticket is proposed to have a simple CLI utility like HadoopKerberosName
> {code}
> hadoop org.apache.hadoop.security.HadoopKerberosName
> nn/localh...@hdpdev.dev.com
> {code}
> The CLI utility for group member lookup will have a usage like below without
> namenode running or restart for configuration change.
> {code}
> hadoop org.apache.hadoop.security.Groups hdfs
> hdfs : [hadoop, hdfs]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13771) Adding group mapping lookup utility without dependency on HDFS namenode
[
https://issues.apache.org/jira/browse/HADOOP-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15617035#comment-15617035
]
Hadoop QA commented on HADOOP-13771:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
2s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13771 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835906/HADOOP-13771.00.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 405735f87bc9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 112f04e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10924/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10924/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Adding group mapping lookup utility without dependency on HDFS namenode
> ---
>
> Key: HADOOP-13771
> URL: https://issues.apache.org/jira/browse/HADOOP-13771
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, tools
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-13771.00.pat
[jira] [Commented] (HADOOP-13771) Adding group mapping lookup utility without dependency on HDFS namenode
[
https://issues.apache.org/jira/browse/HADOOP-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15617033#comment-15617033
]
Allen Wittenauer commented on HADOOP-13771:
---
We are treading on very dangerous ground here.
The reason why the 'hdfs groups' exists is because the NN is considered
authoritative. If the NN and the client are configured differently, then
(historically) the various POSIX commands will give different answers than what
the NN considers true. Now that group resolution is pluggable, the client may
not even have the ability to query the authoritative source!
Removing the ability to query the NN about what it thinks the groups are also
removes a major debugging tool.
There's an additional wrinkle here. The NN is not the only process that is
doing group resolution. Pretty much any service that does ACL resolution also
does group resolution to some degree. Making the command 'hadoop groups' is
going lead some folks to think that this works for any service...
I'd therefore propose a different solution. 'hdfs groups' should work like
nslookup. If the NN is up, it should query the NN and give an authoritative
answer. If the NN is not up, it should give the local answer but be absolutely
clear that it is at best a guess and may be in correct.
> Adding group mapping lookup utility without dependency on HDFS namenode
> ---
>
> Key: HADOOP-13771
> URL: https://issues.apache.org/jira/browse/HADOOP-13771
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, tools
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-13771.00.patch
>
>
> We have {{hdfs groups}} command to troubleshoot issues related to users'
> group member look up with Unix/LDAP. However, there are some limitation of
> this command: 1) it can only be executed when namenode is running. 2) any
> change in the group mapping lookup configuration needs a hdfs namenode
> restart, which is expensive.
> This ticket is proposed to have a simple CLI utility like HadoopKerberosName
> {code}
> hadoop org.apache.hadoop.security.HadoopKerberosName
> nn/localh...@hdpdev.dev.com
> {code}
> The CLI utility for group member lookup will have a usage like below without
> namenode running or restart for configuration change.
> {code}
> hadoop org.apache.hadoop.security.Groups hdfs
> hdfs : [hadoop, hdfs]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13771) Adding group mapping lookup utility without dependency on HDFS namenode
[
https://issues.apache.org/jira/browse/HADOOP-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15616801#comment-15616801
]
Xiaoyu Yao commented on HADOOP-13771:
-
Also propose to migrate existing {{hdfs groups}} command to {{hadoop groups}}
without HDFS dependency.
> Adding group mapping lookup utility without dependency on HDFS namenode
> ---
>
> Key: HADOOP-13771
> URL: https://issues.apache.org/jira/browse/HADOOP-13771
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, tools
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>
> We have {{hdfs groups}} command to troubleshoot issues related to users'
> group member look up with Unix/LDAP. However, there are some limitation of
> this command: 1) it can only be executed when namenode is running. 2) any
> change in the group mapping lookup configuration needs a hdfs namenode
> restart, which is expensive.
> This ticket is proposed to have a simple CLI utility like HadoopKerberosName
> {code}
> hadoop org.apache.hadoop.security.HadoopKerberosName
> nn/localh...@hdpdev.dev.com
> {code}
> The CLI utility for group member lookup will have a usage like below without
> namenode running or restart for configuration change.
> {code}
> hadoop org.apache.hadoop.security.Groups hdfs
> hdfs : [hadoop, hdfs]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
