[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15412853#comment-15412853
]
Bob.zhao commented on YARN-4833:
Hi, Bibin, thanks for you patch!
Here one thing want to know, why not directly throw YarnException directly but
warp with RemoteException?
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch,
> YARN-4833.0002.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
> at org.a
[jira] [Updated] (YARN-5484) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[ https://issues.apache.org/jira/browse/YARN-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob.zhao updated YARN-5484: --- Description: 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 only for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. was: 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. > YARN client will still retry for many times on failover even though RM > server throw AccessControlException > --- > > Key: YARN-5484 > URL: https://issues.apache.org/jira/browse/YARN-5484 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 only for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5484) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[
https://issues.apache.org/jira/browse/YARN-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15411815#comment-15411815
]
Bob.zhao commented on YARN-5484:
I think just like other place on yarn where process AccessControlException to
YarnException , here we can warp AccessControlException to YarnException :
{code}
try {
} catch (AccessControlException e) {
throw new YarnException(e);
}
{code}
Any ideas?
> YARN client will still retry for many times on failover even though RM
> server throw AccessControlException
> ---
>
> Key: YARN-5484
> URL: https://issues.apache.org/jira/browse/YARN-5484
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Bob.zhao
>
> 1. Enable yarn.acl.enable
> 2. Set up some queue for users on yarn, for example queue1 onle for user1
> 3. Using user1 to submit app to queue1, it can run successfully.
> 4. Change user2 to submit app to queue1, it will not be permit submit to
> queue1.
> So, At RM server side, it will throw
> IOException(ClientRMService.java#submitApplication), which is the parent
> implementation of the
> AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this
> IOException will be threw to client that cause yarn client frequently
> failover for many times.
> We'b better avoid this behavior that if client got permission deny from
> server, it should try once and exit, no need to retry.
> This issue was introduced by YARN-4522.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5485) YARN client will still retry for many times on failover even though RM server throw AccessControlException(IOException)
[ https://issues.apache.org/jira/browse/YARN-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15411802#comment-15411802 ] Bob.zhao commented on YARN-5485: rt. sorry to make a mistake for that. let track this issue on YARN-5484. > YARN client will still retry for many times on failover even though RM > server throw AccessControlException(IOException) > > > Key: YARN-5485 > URL: https://issues.apache.org/jira/browse/YARN-5485 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > Issue reproduced steps > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 onle for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5485) YARN client will still retry for many times on failover even though RM server throw AccessControlException(IOException)
[ https://issues.apache.org/jira/browse/YARN-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob.zhao updated YARN-5485: --- Summary: YARN client will still retry for many times on failover even though RM server throw AccessControlException(IOException) (was: YARN client will still retry for many times on failover even though RM server throw AccessControlException) > YARN client will still retry for many times on failover even though RM > server throw AccessControlException(IOException) > > > Key: YARN-5485 > URL: https://issues.apache.org/jira/browse/YARN-5485 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > Issue reproduced steps > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 onle for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5485) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[ https://issues.apache.org/jira/browse/YARN-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob.zhao updated YARN-5485: --- Description: Issue reproduced steps 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. was: 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. > YARN client will still retry for many times on failover even though RM > server throw AccessControlException > --- > > Key: YARN-5485 > URL: https://issues.apache.org/jira/browse/YARN-5485 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > Issue reproduced steps > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 onle for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5485) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[ https://issues.apache.org/jira/browse/YARN-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob.zhao updated YARN-5485: --- Description: 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. was: 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. > YARN client will still retry for many times on failover even though RM > server throw AccessControlException > --- > > Key: YARN-5485 > URL: https://issues.apache.org/jira/browse/YARN-5485 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 onle for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5485) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[ https://issues.apache.org/jira/browse/YARN-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob.zhao updated YARN-5485: --- Description: 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. was: 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. > YARN client will still retry for many times on failover even though RM > server throw AccessControlException > --- > > Key: YARN-5485 > URL: https://issues.apache.org/jira/browse/YARN-5485 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 onle for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), > this IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5484) YARN client will still retry for many times on failover even though RM server throw AccessControlException
Bob.zhao created YARN-5484: -- Summary: YARN client will still retry for many times on failover even though RM server throw AccessControlException Key: YARN-5484 URL: https://issues.apache.org/jira/browse/YARN-5484 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.9.0 Reporter: Bob.zhao 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5485) YARN client will still retry for many times on failover even though RM server throw AccessControlException
Bob.zhao created YARN-5485: -- Summary: YARN client will still retry for many times on failover even though RM server throw AccessControlException Key: YARN-5485 URL: https://issues.apache.org/jira/browse/YARN-5485 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.9.0 Reporter: Bob.zhao 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
