[jira] [Commented] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129072#comment-14129072
]
Hitesh Shah commented on TEZ-1524:
--
Comment on patch:
{code}
+ACLManager aclManager = real.getACLManager();
+if (aclManager.isEnabled()) {
+ String user = getRPCUserName();
+ if (!real.getACLManager().checkAMViewAccess(user, getRPCUserGroups())) {
+throw new AccessControlException(User + user
++ cannot perform AM view operation);
+ }
{code}
Instead of above, it might be better to pass the UGI into the acl manager
function call i.e. real.getACLManager().checkAMViewAccess(currentUserUGI) .
getDAGStatus seems to fork out the entire JVM
-
Key: TEZ-1524
URL: https://issues.apache.org/jira/browse/TEZ-1524
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Gopal V
Assignee: Gopal V
Attachments: TEZ-1524.1.patch
Tracked down a consistent fork() call to
{code}
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
at
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
at
org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
{code}
[~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115774#comment-14115774
]
Hitesh Shah commented on TEZ-1524:
--
Nevermind, ran a simple job on a single node cluster. The AM logs show fetched
groups logged once followed by cached groups.
getDAGStatus seems to fork out the entire JVM
-
Key: TEZ-1524
URL: https://issues.apache.org/jira/browse/TEZ-1524
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.6.0
Reporter: Gopal V
Fix For: 0.6.0
Tracked down a consistent fork() call to
{code}
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
at
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
at
org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
{code}
[~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115781#comment-14115781
]
Hitesh Shah commented on TEZ-1524:
--
[~gopalv] Could you add more details as this points to a bug in UGI if it is
indeed forking on each ask for groupNames.
getDAGStatus seems to fork out the entire JVM
-
Key: TEZ-1524
URL: https://issues.apache.org/jira/browse/TEZ-1524
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.6.0
Reporter: Gopal V
Fix For: 0.6.0
Tracked down a consistent fork() call to
{code}
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
at
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
at
org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
{code}
[~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115806#comment-14115806
]
Gopal V commented on TEZ-1524:
--
The cache does not cache misses.
getDAGStatus seems to fork out the entire JVM
-
Key: TEZ-1524
URL: https://issues.apache.org/jira/browse/TEZ-1524
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Gopal V
Tracked down a consistent fork() call to
{code}
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
at
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
at
org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
{code}
[~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115877#comment-14115877
]
Hitesh Shah commented on TEZ-1524:
--
Is this in a non-secure cluster with acls disabled? We might be able to work
around this to invoking getGroups only when needed. It wont prevent calls for
non-existent users but should help a bit.
getDAGStatus seems to fork out the entire JVM
-
Key: TEZ-1524
URL: https://issues.apache.org/jira/browse/TEZ-1524
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Gopal V
Attachments: TEZ-1524.1.patch
Tracked down a consistent fork() call to
{code}
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
at
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
at
org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
{code}
[~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
