[jira] [Commented] (YARN-6526) Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call
[
https://issues.apache.org/jira/browse/YARN-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097545#comment-17097545
]
Giovanni Matteo Fumarola commented on YARN-6526:
Currently the code get a connection from HikariPool and close the connections
at the end of the query.
{code:java}
conn = getConnection();
cstmt = conn.prepareCall(CALL_SP_REGISTER_SUBCLUSTER);
FederationStateStoreUtils.returnToPool(LOG, cstmt, conn);{code}
I would like to see if we can remove the cstmt creation and deletion. I don't
know if it is possible or not.
If not, we can close the jira.
> Refactoring SQLFederationStateStore by avoiding to recreate a connection at
> every call
> --
>
> Key: YARN-6526
> URL: https://issues.apache.org/jira/browse/YARN-6526
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Reporter: Giovanni Matteo Fumarola
>Assignee: Bilwa S T
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6526) Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call
[ https://issues.apache.org/jira/browse/YARN-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-6526: --- Summary: Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call (was: Refactoring SQLFederationStateStore by avoiding to recreate the connections at every call) > Refactoring SQLFederationStateStore by avoiding to recreate a connection at > every call > -- > > Key: YARN-6526 > URL: https://issues.apache.org/jira/browse/YARN-6526 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Giovanni Matteo Fumarola >Assignee: Bilwa S T >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6924) Metrics for Federation AMRMProxy
[ https://issues.apache.org/jira/browse/YARN-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17049876#comment-17049876 ] Giovanni Matteo Fumarola commented on YARN-6924: +1 on [^YARN-6924.05.patch] > Metrics for Federation AMRMProxy > > > Key: YARN-6924 > URL: https://issues.apache.org/jira/browse/YARN-6924 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Young Chen >Priority: Major > Attachments: YARN-6924.01.patch, YARN-6924.01.patch, > YARN-6924.02.patch, YARN-6924.02.patch, YARN-6924.03.patch, > YARN-6924.04.patch, YARN-6924.05.patch > > > This JIRA proposes addition of metrics for Federation AMRMProxy -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8982) [Router] Add locality policy
[ https://issues.apache.org/jira/browse/YARN-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027112#comment-17027112 ] Giovanni Matteo Fumarola commented on YARN-8982: Thanks [~youchen] for picking up one of my old patches and fix several minor things. Committed to trunk. > [Router] Add locality policy > - > > Key: YARN-8982 > URL: https://issues.apache.org/jira/browse/YARN-8982 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Young Chen >Priority: Major > Attachments: YARN-8982.v1.patch, YARN-8982.v2.patch, > YARN-8982.v3.patch, YARN-8982.v4.patch, YARN-8982.v5.patch > > > This jira tracks the effort to add a new policy in the Router. > This policy will allow the Router to pick the SubCluster based on the node > that the client requested. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8982) [Router] Add locality policy
[ https://issues.apache.org/jira/browse/YARN-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-8982: --- Fix Version/s: 3.4.0 > [Router] Add locality policy > - > > Key: YARN-8982 > URL: https://issues.apache.org/jira/browse/YARN-8982 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Young Chen >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-8982.v1.patch, YARN-8982.v2.patch, > YARN-8982.v3.patch, YARN-8982.v4.patch, YARN-8982.v5.patch > > > This jira tracks the effort to add a new policy in the Router. > This policy will allow the Router to pick the SubCluster based on the node > that the client requested. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10038) [UI] Finish Time is not correctly parsed in the RM Apps page
[ https://issues.apache.org/jira/browse/YARN-10038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-10038: Component/s: webapp > [UI] Finish Time is not correctly parsed in the RM Apps page > > > Key: YARN-10038 > URL: https://issues.apache.org/jira/browse/YARN-10038 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp >Affects Versions: 3.3.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-10038.000.patch, YARN-10038.001.patch, > YARN-10038.002.patch, YARN-10038.003.patch, image-2019-12-17-11-08-22-026.png > > > The Finish Time shows as the unix time (millis since 1970) instead of as a > date: > !image-2019-12-17-11-08-22-026.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10038) [UI] Finish Time is not correctly parsed in the RM Apps page
[ https://issues.apache.org/jira/browse/YARN-10038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-10038: Affects Version/s: 3.3.0 > [UI] Finish Time is not correctly parsed in the RM Apps page > > > Key: YARN-10038 > URL: https://issues.apache.org/jira/browse/YARN-10038 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-10038.000.patch, YARN-10038.001.patch, > YARN-10038.002.patch, YARN-10038.003.patch, image-2019-12-17-11-08-22-026.png > > > The Finish Time shows as the unix time (millis since 1970) instead of as a > date: > !image-2019-12-17-11-08-22-026.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10038) [UI] Finish Time is not correctly parsed in the RM Apps page
[ https://issues.apache.org/jira/browse/YARN-10038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-10038: Priority: Minor (was: Major) > [UI] Finish Time is not correctly parsed in the RM Apps page > > > Key: YARN-10038 > URL: https://issues.apache.org/jira/browse/YARN-10038 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-10038.000.patch, YARN-10038.001.patch, > YARN-10038.002.patch, YARN-10038.003.patch, image-2019-12-17-11-08-22-026.png > > > The Finish Time shows as the unix time (millis since 1970) instead of as a > date: > !image-2019-12-17-11-08-22-026.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10038) [UI] Finish Time is not correctly parsed in the RM Apps page
[ https://issues.apache.org/jira/browse/YARN-10038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-10038: Fix Version/s: 3.3.0 > [UI] Finish Time is not correctly parsed in the RM Apps page > > > Key: YARN-10038 > URL: https://issues.apache.org/jira/browse/YARN-10038 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10038.000.patch, YARN-10038.001.patch, > YARN-10038.002.patch, YARN-10038.003.patch, image-2019-12-17-11-08-22-026.png > > > The Finish Time shows as the unix time (millis since 1970) instead of as a > date: > !image-2019-12-17-11-08-22-026.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10038) [UI] Finish Time is not correctly parsed in the RM Apps page
[ https://issues.apache.org/jira/browse/YARN-10038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17000324#comment-17000324 ] Giovanni Matteo Fumarola commented on YARN-10038: - Committed to trunk [^YARN-10038.003.patch]. Thanks [~elgoiri]. > [UI] Finish Time is not correctly parsed in the RM Apps page > > > Key: YARN-10038 > URL: https://issues.apache.org/jira/browse/YARN-10038 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: YARN-10038.000.patch, YARN-10038.001.patch, > YARN-10038.002.patch, YARN-10038.003.patch, image-2019-12-17-11-08-22-026.png > > > The Finish Time shows as the unix time (millis since 1970) instead of as a > date: > !image-2019-12-17-11-08-22-026.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10038) [UI] Finish Time is not correctly parsed in the RM Apps page
[ https://issues.apache.org/jira/browse/YARN-10038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999442#comment-16999442 ] Giovanni Matteo Fumarola commented on YARN-10038: - Thanks [~elgoiri] for working on this and add a unit test that will prevent this to happen again. I do not see any issue with [^YARN-10038.002.patch] . We can commit this after Yetus results. > [UI] Finish Time is not correctly parsed in the RM Apps page > > > Key: YARN-10038 > URL: https://issues.apache.org/jira/browse/YARN-10038 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: YARN-10038.000.patch, YARN-10038.001.patch, > YARN-10038.002.patch, image-2019-12-17-11-08-22-026.png > > > The Finish Time shows as the unix time (millis since 1970) instead of as a > date: > !image-2019-12-17-11-08-22-026.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9689) Router does not support kerberos proxy when in secure mode
[
https://issues.apache.org/jira/browse/YARN-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967131#comment-16967131
]
Giovanni Matteo Fumarola commented on YARN-9689:
Thanks [~cane] for the patch and [~botong] for the review.
The patch looks good. Approved the PR.
> Router does not support kerberos proxy when in secure mode
> --
>
> Key: YARN-9689
> URL: https://issues.apache.org/jira/browse/YARN-9689
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.1.2
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9689.001.patch
>
>
> When we enable kerberos in YARN-Federation mode, we can not get new app since
> it will throw kerberos exception below.Which should be handled!
> {code:java}
> 2019-07-22,18:43:25,523 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 2019-07-22,18:43:25,528 WARN
> org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor:
> Unable to create a new ApplicationId in SubCluster xxx
> java.io.IOException: DestHost:destPort xxx , LocalHost:localPort xxx. Failed
> on local exception: java.io.IOException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1564)
> at org.apache.hadoop.ipc.Client.call(Client.java:1506)
> at org.apache.hadoop.ipc.Client.call(Client.java:1416)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy91.getNewApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:274)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy92.getNewApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.getNewApplication(FederationClientInterceptor.java:252)
> at
> org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.getNewApplication(RouterClientRMService.java:218)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getNewApplication(ApplicationClientProtocolPBServiceImpl.java:263)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:559)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:525)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:992)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:885)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:831)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
[jira] [Resolved] (YARN-9689) Router does not support kerberos proxy when in secure mode
[
https://issues.apache.org/jira/browse/YARN-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola resolved YARN-9689.
Resolution: Fixed
> Router does not support kerberos proxy when in secure mode
> --
>
> Key: YARN-9689
> URL: https://issues.apache.org/jira/browse/YARN-9689
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.1.2
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9689.001.patch
>
>
> When we enable kerberos in YARN-Federation mode, we can not get new app since
> it will throw kerberos exception below.Which should be handled!
> {code:java}
> 2019-07-22,18:43:25,523 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 2019-07-22,18:43:25,528 WARN
> org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor:
> Unable to create a new ApplicationId in SubCluster xxx
> java.io.IOException: DestHost:destPort xxx , LocalHost:localPort xxx. Failed
> on local exception: java.io.IOException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1564)
> at org.apache.hadoop.ipc.Client.call(Client.java:1506)
> at org.apache.hadoop.ipc.Client.call(Client.java:1416)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy91.getNewApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:274)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy92.getNewApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.getNewApplication(FederationClientInterceptor.java:252)
> at
> org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.getNewApplication(RouterClientRMService.java:218)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getNewApplication(ApplicationClientProtocolPBServiceImpl.java:263)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:559)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:525)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:992)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:885)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:831)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1716)
> at
[jira] [Commented] (YARN-9874) Remove unnecessary LevelDb write call in LeveldbConfigurationStore#confirmMutation
[
https://issues.apache.org/jira/browse/YARN-9874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946145#comment-16946145
]
Giovanni Matteo Fumarola commented on YARN-9874:
Thanks [~Prabhu Joseph].
I think there is an extra file in the patch.
> Remove unnecessary LevelDb write call in
> LeveldbConfigurationStore#confirmMutation
> --
>
> Key: YARN-9874
> URL: https://issues.apache.org/jira/browse/YARN-9874
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9874-001.patch
>
>
> Remove unnecessary LevelDb write call in
> LeveldbConfigurationStore#confirmMutation.
> {code}
> public void confirmMutation(boolean isValid) throws IOException {
> WriteBatch updateBatch = db.createWriteBatch();
> if (isValid) {
> ...
> }
> db.write(updateBatch);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6740) Federation Router (hiding multiple RMs for ApplicationClientProtocol) phase 2
[ https://issues.apache.org/jira/browse/YARN-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875094#comment-16875094 ] Giovanni Matteo Fumarola commented on YARN-6740: Only 6-7 methods got implemented in the several jiras. There are still methods that need to be implemented. > Federation Router (hiding multiple RMs for ApplicationClientProtocol) phase 2 > - > > Key: YARN-6740 > URL: https://issues.apache.org/jira/browse/YARN-6740 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Abhishek Modi >Priority: Major > > This JIRA tracks the implementation of the layer for routing > ApplicaitonClientProtocol requests to the appropriate RM(s) in a federated > YARN cluster. > Under the YARN-3659 we only implemented getNewApplication, submitApplication, > forceKillApplication and getApplicationReport to execute applications E2E. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6055) ContainersMonitorImpl need be adjusted when NM resource changed.
[ https://issues.apache.org/jira/browse/YARN-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-6055: --- Fix Version/s: 3.3.0 > ContainersMonitorImpl need be adjusted when NM resource changed. > > > Key: YARN-6055 > URL: https://issues.apache.org/jira/browse/YARN-6055 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Íñigo Goiri >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-6055.000.patch, YARN-6055.001.patch, > YARN-6055.002.patch, YARN-6055.003.patch, YARN-6055.004.patch > > > Per Ravi's comments in YARN-4832, we need to check some limits in > containerMonitorImpl to make sure it get updated also when Resource updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6055) ContainersMonitorImpl need be adjusted when NM resource changed.
[ https://issues.apache.org/jira/browse/YARN-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873655#comment-16873655 ] Giovanni Matteo Fumarola commented on YARN-6055: Thanks [~elgoiri] for the patch. Looks good to me and the refactor helps the readability. Committed to trunk. > ContainersMonitorImpl need be adjusted when NM resource changed. > > > Key: YARN-6055 > URL: https://issues.apache.org/jira/browse/YARN-6055 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Íñigo Goiri >Priority: Major > Attachments: YARN-6055.000.patch, YARN-6055.001.patch, > YARN-6055.002.patch, YARN-6055.003.patch, YARN-6055.004.patch > > > Per Ravi's comments in YARN-4832, we need to check some limits in > containerMonitorImpl to make sure it get updated also when Resource updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9599) TestContainerSchedulerQueuing#testQueueShedding fails intermittently.
[ https://issues.apache.org/jira/browse/YARN-9599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863345#comment-16863345 ] Giovanni Matteo Fumarola commented on YARN-9599: Committed to trunk. Thanks [~elgoiri] for the review and [~abmodi] for the patch. > TestContainerSchedulerQueuing#testQueueShedding fails intermittently. > - > > Key: YARN-9599 > URL: https://issues.apache.org/jira/browse/YARN-9599 > Project: Hadoop YARN > Issue Type: Task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9599.001.patch, YARN-9599.002.patch, > YARN-9599.003.patch, YARN-9599.004.patch > > > TestQueueShedding fails intermittently. > java.lang.AssertionError: expected:<6> but was:<5> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at org.junit.Assert.assertEquals(Assert.java:631) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing.testQueueShedding(TestContainerSchedulerQueuing.java:775) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) > > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) > > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) > at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9599) TestContainerSchedulerQueuing#testQueueShedding fails intermittently.
[ https://issues.apache.org/jira/browse/YARN-9599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9599: --- Fix Version/s: 3.3.0 > TestContainerSchedulerQueuing#testQueueShedding fails intermittently. > - > > Key: YARN-9599 > URL: https://issues.apache.org/jira/browse/YARN-9599 > Project: Hadoop YARN > Issue Type: Task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9599.001.patch, YARN-9599.002.patch, > YARN-9599.003.patch, YARN-9599.004.patch > > > TestQueueShedding fails intermittently. > java.lang.AssertionError: expected:<6> but was:<5> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at org.junit.Assert.assertEquals(Assert.java:631) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing.testQueueShedding(TestContainerSchedulerQueuing.java:775) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) > > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) > > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) > at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9592) Use Logger format in ContainersMonitorImpl
[
https://issues.apache.org/jira/browse/YARN-9592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853510#comment-16853510
]
Giovanni Matteo Fumarola commented on YARN-9592:
Thanks [~elgoiri] for working on this.
Committed to trunk.
We should do this work for several other classes.
> Use Logger format in ContainersMonitorImpl
> --
>
> Key: YARN-9592
> URL: https://issues.apache.org/jira/browse/YARN-9592
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9592.000.patch, YARN-9592.001.patch
>
>
> {{ContainersMonitorImpl}} currently has Logger but it is not taking advantage
> of the replacements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9592) Use Logger format in ContainersMonitorImpl
[
https://issues.apache.org/jira/browse/YARN-9592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9592:
---
Fix Version/s: 3.3.0
> Use Logger format in ContainersMonitorImpl
> --
>
> Key: YARN-9592
> URL: https://issues.apache.org/jira/browse/YARN-9592
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9592.000.patch, YARN-9592.001.patch
>
>
> {{ContainersMonitorImpl}} currently has Logger but it is not taking advantage
> of the replacements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9425) Make initialDelay configurable for FederationStateStoreService#scheduledExecutorService
[ https://issues.apache.org/jira/browse/YARN-9425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853502#comment-16853502 ] Giovanni Matteo Fumarola commented on YARN-9425: Thanks [~shenyinjie] for working on this. NIT: Looks like the patch has a different coding style from the class. 1) It should be "public static final String FEDERATION_STATESTORE_HEARTBEAT_INITIAL_DELAY_SECS =" 2) YarnConfiguration.FEDERATION_STATESTORE_HEARTBEAT_INITIAL_DELAY_SECS should go in the same line. 3) The description in yarn-default should be "Initial delay for federation state-store heartbeat service" > Make initialDelay configurable for > FederationStateStoreService#scheduledExecutorService > --- > > Key: YARN-9425 > URL: https://issues.apache.org/jira/browse/YARN-9425 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Affects Versions: 3.1.0 >Reporter: Shen Yinjie >Assignee: Shen Yinjie >Priority: Major > Attachments: YARN-9425_1.patch, YARN-9425_2.patch, YARN-9425_3.patch > > > When enable YARN federation, subclusters info in Router Web UI cannot be > loaded immediately, and client cannot find any active subclusters after 5mins > by default ,which is configured by > "yarn.federation.state-store.heartbeat-interval-secs". > IMA,we should seperate 'initialDely' and 'delay' for > FederationStateStoreService#scheduledExecutorService. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8199) Logging fileSize of log files under NM Local Dir
[ https://issues.apache.org/jira/browse/YARN-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853497#comment-16853497 ] Giovanni Matteo Fumarola commented on YARN-8199: Thanks [~Prabhu Joseph] for the patch. I think a limit should be done for the logs file not only for the debug. > Logging fileSize of log files under NM Local Dir > > > Key: YARN-8199 > URL: https://issues.apache.org/jira/browse/YARN-8199 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: supportability > Attachments: 0001-YARN-8199.patch, 0002-YARN-8199.patch, > YARN-8199-003.patch > > > Logging fileSize of log files like syslog, stderr, stdout under NM Local Dir > by NodeManager before the cleanup will help to find the application which has > written too verbose. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9592) Use Logger format in ContainersMonitorImpl
[
https://issues.apache.org/jira/browse/YARN-9592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852456#comment-16852456
]
Giovanni Matteo Fumarola commented on YARN-9592:
Thanks [~elgoiri] for the patch.
Good catch on tmp.append() at line 471.
NIT: Line 141 needs {} and Line 513 needs an extra space.
> Use Logger format in ContainersMonitorImpl
> --
>
> Key: YARN-9592
> URL: https://issues.apache.org/jira/browse/YARN-9592
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: YARN-9592.000.patch
>
>
> {{ContainersMonitorImpl}} currently has Logger but it is not taking advantage
> of the replacements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9553) Fix NPE in EntityGroupFSTimelineStore#getEntityTimelines
[
https://issues.apache.org/jira/browse/YARN-9553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9553:
---
Fix Version/s: 3.3.0
> Fix NPE in EntityGroupFSTimelineStore#getEntityTimelines
>
>
> Key: YARN-9553
> URL: https://issues.apache.org/jira/browse/YARN-9553
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9553-001.patch
>
>
> TimelineWebService Rest Api entityType/events throws NPE
> {code}
> http://172.25.35.146:8188/ws/v1/timeline/YARN_APPLICATION/events
> {"exception":"WebApplicationException","message":"java.lang.NullPointerException","javaClassName":"javax.ws.rs.WebApplicationException"}
> 2019-05-14 11:47:38,376 WARN webapp.GenericExceptionHandler
> (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR
> javax.ws.rs.WebApplicationException: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEvents(TimelineWebServices.java:206)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.security.http.RestCsrfPreventionFilter$ServletFilterHttpInteraction.proceed(RestCsrfPreventionFilter.java:269)
> at
> org.apache.hadoop.security.http.RestCsrfPreventionFilter.handleHttpInteraction(RestCsrfPreventionFilter.java:197)
> at
> org.apache.hadoop.security.http.RestCsrfPreventionFilter.doFilter(RestCsrfPreventionFilter.java:209)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
> at
>
[jira] [Commented] (YARN-9553) Fix NPE in EntityGroupFSTimelineStore#getEntityTimelines
[
https://issues.apache.org/jira/browse/YARN-9553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852233#comment-16852233
]
Giovanni Matteo Fumarola commented on YARN-9553:
Thanks [~Prabhu Joseph]. The fix is straight-forward. Committed to trunk.
> Fix NPE in EntityGroupFSTimelineStore#getEntityTimelines
>
>
> Key: YARN-9553
> URL: https://issues.apache.org/jira/browse/YARN-9553
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9553-001.patch
>
>
> TimelineWebService Rest Api entityType/events throws NPE
> {code}
> http://172.25.35.146:8188/ws/v1/timeline/YARN_APPLICATION/events
> {"exception":"WebApplicationException","message":"java.lang.NullPointerException","javaClassName":"javax.ws.rs.WebApplicationException"}
> 2019-05-14 11:47:38,376 WARN webapp.GenericExceptionHandler
> (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR
> javax.ws.rs.WebApplicationException: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEvents(TimelineWebServices.java:206)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.security.http.RestCsrfPreventionFilter$ServletFilterHttpInteraction.proceed(RestCsrfPreventionFilter.java:269)
> at
> org.apache.hadoop.security.http.RestCsrfPreventionFilter.handleHttpInteraction(RestCsrfPreventionFilter.java:197)
> at
> org.apache.hadoop.security.http.RestCsrfPreventionFilter.doFilter(RestCsrfPreventionFilter.java:209)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
>
[jira] [Updated] (YARN-9553) Fix NPE in EntityGroupFSTimelineStore#getEntityTimelines
[
https://issues.apache.org/jira/browse/YARN-9553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9553:
---
Summary: Fix NPE in EntityGroupFSTimelineStore#getEntityTimelines (was:
Fix NPE from TimelineWebService getEvents)
> Fix NPE in EntityGroupFSTimelineStore#getEntityTimelines
>
>
> Key: YARN-9553
> URL: https://issues.apache.org/jira/browse/YARN-9553
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9553-001.patch
>
>
> TimelineWebService Rest Api entityType/events throws NPE
> {code}
> http://172.25.35.146:8188/ws/v1/timeline/YARN_APPLICATION/events
> {"exception":"WebApplicationException","message":"java.lang.NullPointerException","javaClassName":"javax.ws.rs.WebApplicationException"}
> 2019-05-14 11:47:38,376 WARN webapp.GenericExceptionHandler
> (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR
> javax.ws.rs.WebApplicationException: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEvents(TimelineWebServices.java:206)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.security.http.RestCsrfPreventionFilter$ServletFilterHttpInteraction.proceed(RestCsrfPreventionFilter.java:269)
> at
> org.apache.hadoop.security.http.RestCsrfPreventionFilter.handleHttpInteraction(RestCsrfPreventionFilter.java:197)
> at
> org.apache.hadoop.security.http.RestCsrfPreventionFilter.doFilter(RestCsrfPreventionFilter.java:209)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
> at
>
[jira] [Commented] (YARN-9482) DistributedShell job with localization fails in unsecure cluster
[
https://issues.apache.org/jira/browse/YARN-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847036#comment-16847036
]
Giovanni Matteo Fumarola commented on YARN-9482:
Thanks [~pbacsko] and [~sunilg] for the initial review and [~Prabhu Joseph] for
the patch.
The fix seems to do the right thing by changing the path of the home directory.
Committed to trunk.
> DistributedShell job with localization fails in unsecure cluster
>
>
> Key: YARN-9482
> URL: https://issues.apache.org/jira/browse/YARN-9482
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-shell
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9482-001.patch, YARN-9482-002.patch,
> YARN-9482-003.patch, YARN-9482-004.patch
>
>
> DistributedShell job with localization fails in unsecure cluster. The client
> localizes the input files to home directory (job user) whereas the AM runs as
> yarn user reads from it's home directory.
> *Command:*
> {code}
> yarn jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -shell_command ls -shell_args / -jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -localize_files /tmp/prabhu
> {code}
> {code}
> Exception in thread "Thread-4" java.io.UncheckedIOException: Error during
> localization setup
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1495)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.run(ApplicationMaster.java:1481)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs://yarn-ats-1:8020/user/yarn/DistributedShell/application_1554817981283_0003/prabhu
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1586)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1594)
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1487)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9482) DistributedShell job with localization fails in unsecure cluster
[
https://issues.apache.org/jira/browse/YARN-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9482:
---
Fix Version/s: 3.3.0
> DistributedShell job with localization fails in unsecure cluster
>
>
> Key: YARN-9482
> URL: https://issues.apache.org/jira/browse/YARN-9482
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-shell
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9482-001.patch, YARN-9482-002.patch,
> YARN-9482-003.patch, YARN-9482-004.patch
>
>
> DistributedShell job with localization fails in unsecure cluster. The client
> localizes the input files to home directory (job user) whereas the AM runs as
> yarn user reads from it's home directory.
> *Command:*
> {code}
> yarn jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -shell_command ls -shell_args / -jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -localize_files /tmp/prabhu
> {code}
> {code}
> Exception in thread "Thread-4" java.io.UncheckedIOException: Error during
> localization setup
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1495)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.run(ApplicationMaster.java:1481)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs://yarn-ats-1:8020/user/yarn/DistributedShell/application_1554817981283_0003/prabhu
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1586)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1594)
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1487)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9482) DistributedShell job with localization fails in unsecure cluster
[
https://issues.apache.org/jira/browse/YARN-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846926#comment-16846926
]
Giovanni Matteo Fumarola commented on YARN-9482:
Can we re-run yetus for the updated results?
> DistributedShell job with localization fails in unsecure cluster
>
>
> Key: YARN-9482
> URL: https://issues.apache.org/jira/browse/YARN-9482
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-shell
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9482-001.patch, YARN-9482-002.patch,
> YARN-9482-003.patch
>
>
> DistributedShell job with localization fails in unsecure cluster. The client
> localizes the input files to home directory (job user) whereas the AM runs as
> yarn user reads from it's home directory.
> *Command:*
> {code}
> yarn jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -shell_command ls -shell_args / -jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -localize_files /tmp/prabhu
> {code}
> {code}
> Exception in thread "Thread-4" java.io.UncheckedIOException: Error during
> localization setup
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1495)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.run(ApplicationMaster.java:1481)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs://yarn-ats-1:8020/user/yarn/DistributedShell/application_1554817981283_0003/prabhu
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1586)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1594)
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1487)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9488) Skip YARNFeatureNotEnabledException from ClientRMService
[
https://issues.apache.org/jira/browse/YARN-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846925#comment-16846925
]
Giovanni Matteo Fumarola commented on YARN-9488:
Can we re-run Yetus for the updated result?
> Skip YARNFeatureNotEnabledException from ClientRMService
>
>
> Key: YARN-9488
> URL: https://issues.apache.org/jira/browse/YARN-9488
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9488-001.patch
>
>
> RM logs are accumulated with YARNFeatureNotEnabledException when running
> DIstributed Shell jobs while {{ClientRMService#getResourceProfiles}}
> {code}
> 2019-04-16 07:10:47,699 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 0 on 8050, call Call#5 Retry#0
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getResourceProfiles
> from 172.26.81.91:41198
> org.apache.hadoop.yarn.exceptions.YARNFeatureNotEnabledException: Resource
> profile is not enabled, please enable resource profile feature before using
> its functions. (by setting yarn.resourcemanager.resource-profiles.enabled to
> true)
> at
> org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.checkAndThrowExceptionWhenFeatureDisabled(ResourceProfilesManagerImpl.java:191)
> at
> org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.getResourceProfiles(ResourceProfilesManagerImpl.java:214)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getResourceProfiles(ClientRMService.java:1833)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getResourceProfiles(ApplicationClientProtocolPBServiceImpl.java:670)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:665)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9575) Fix TestYarnConfigurationFields testcase failing
[
https://issues.apache.org/jira/browse/YARN-9575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9575:
---
Fix Version/s: 3.3.0
> Fix TestYarnConfigurationFields testcase failing
>
>
> Key: YARN-9575
> URL: https://issues.apache.org/jira/browse/YARN-9575
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, yarn-native-services
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9575-001.patch
>
>
> TestYarnConfigurationFields>TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass
> is failing
> {code}
> [ERROR] testCompareXmlAgainstConfigurationClass Time elapsed: 0.244 s <<<
> FAILURE!
> java.lang.AssertionError: yarn-default.xml has 1 properties missing in class
> org.apache.hadoop.yarn.conf.YarnConfiguration Entries:
> yarn.service.classpath expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass(TestConfigurationFieldsBase.java:540)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:39)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:79)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:70)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
> at
> org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
> at
> org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
> at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:142)
> at
>
[jira] [Comment Edited] (YARN-9575) Fix TestYarnConfigurationFields testcase failing
[
https://issues.apache.org/jira/browse/YARN-9575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845114#comment-16845114
]
Giovanni Matteo Fumarola edited comment on YARN-9575 at 5/21/19 6:26 PM:
-
The test failure was listed here :
[https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1143/testReport/junit/org.apache.hadoop.yarn.conf/TestYarnConfigurationFields/testCompareXmlAgainstConfigurationClass/]
it was due to YARN-9546.
Committed to trunk. Thanks [~Prabhu Joseph]
was (Author: giovanni.fumarola):
The test failure was listed here :
[https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1143/testReport/junit/org.apache.hadoop.yarn.conf/TestYarnConfigurationFields/testCompareXmlAgainstConfigurationClass/]
it was due to YARN-9546.
Committing to trunk.
> Fix TestYarnConfigurationFields testcase failing
>
>
> Key: YARN-9575
> URL: https://issues.apache.org/jira/browse/YARN-9575
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, yarn-native-services
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9575-001.patch
>
>
> TestYarnConfigurationFields>TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass
> is failing
> {code}
> [ERROR] testCompareXmlAgainstConfigurationClass Time elapsed: 0.244 s <<<
> FAILURE!
> java.lang.AssertionError: yarn-default.xml has 1 properties missing in class
> org.apache.hadoop.yarn.conf.YarnConfiguration Entries:
> yarn.service.classpath expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass(TestConfigurationFieldsBase.java:540)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:39)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:79)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:70)
> at
>
[jira] [Updated] (YARN-9575) Fix TestYarnConfigurationFields testcase failing
[
https://issues.apache.org/jira/browse/YARN-9575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9575:
---
Summary: Fix TestYarnConfigurationFields testcase failing (was:
TestYarnConfigurationFields testcase failing)
> Fix TestYarnConfigurationFields testcase failing
>
>
> Key: YARN-9575
> URL: https://issues.apache.org/jira/browse/YARN-9575
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, yarn-native-services
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9575-001.patch
>
>
> TestYarnConfigurationFields>TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass
> is failing
> {code}
> [ERROR] testCompareXmlAgainstConfigurationClass Time elapsed: 0.244 s <<<
> FAILURE!
> java.lang.AssertionError: yarn-default.xml has 1 properties missing in class
> org.apache.hadoop.yarn.conf.YarnConfiguration Entries:
> yarn.service.classpath expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass(TestConfigurationFieldsBase.java:540)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:39)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:79)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:70)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
> at
> org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
> at
> org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
> at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:142)
> at
>
[jira] [Commented] (YARN-9575) TestYarnConfigurationFields testcase failing
[
https://issues.apache.org/jira/browse/YARN-9575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845114#comment-16845114
]
Giovanni Matteo Fumarola commented on YARN-9575:
The test failure was listed here :
[https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1143/testReport/junit/org.apache.hadoop.yarn.conf/TestYarnConfigurationFields/testCompareXmlAgainstConfigurationClass/]
it was due to YARN-9546.
Committing to trunk.
> TestYarnConfigurationFields testcase failing
>
>
> Key: YARN-9575
> URL: https://issues.apache.org/jira/browse/YARN-9575
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, yarn-native-services
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9575-001.patch
>
>
> TestYarnConfigurationFields>TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass
> is failing
> {code}
> [ERROR] testCompareXmlAgainstConfigurationClass Time elapsed: 0.244 s <<<
> FAILURE!
> java.lang.AssertionError: yarn-default.xml has 1 properties missing in class
> org.apache.hadoop.yarn.conf.YarnConfiguration Entries:
> yarn.service.classpath expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass(TestConfigurationFieldsBase.java:540)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:39)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:79)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:70)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
> at
> org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
> at
> org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
> at
>
[jira] [Commented] (YARN-9543) UI2 should handle missing ATSv2 gracefully
[ https://issues.apache.org/jira/browse/YARN-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844252#comment-16844252 ] Giovanni Matteo Fumarola commented on YARN-9543: [~wangda], we do not have the same request internally. I am just helping with the review :) Disable the flows tab seems the right approach. > UI2 should handle missing ATSv2 gracefully > -- > > Key: YARN-9543 > URL: https://issues.apache.org/jira/browse/YARN-9543 > Project: Hadoop YARN > Issue Type: Improvement > Components: ATSv2, yarn-ui-v2 >Affects Versions: 3.1.2 >Reporter: Zoltan Siegl >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9543.001.patch > > > Resource manager UI2 is throwing some console errors and a error page on > flows page. > Suggested improvements: > * Disable or remove flows tab if ATSv2 is not available/installed > * Handle all connection errors to ATSv2 gracefully -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9563) Resource report REST API could return NaN or Inf
[ https://issues.apache.org/jira/browse/YARN-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842406#comment-16842406 ] Giovanni Matteo Fumarola commented on YARN-9563: Thanks [~ahussein]. I would avoid writing comments on the caller methods. I am not sure adding an extra check in Protob class is the correct fix. > Resource report REST API could return NaN or Inf > > > Key: YARN-9563 > URL: https://issues.apache.org/jira/browse/YARN-9563 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: YARN-9563.001.patch > > > The Resource Manager's Cluster Applications and Cluster Application REST APIs > are sometimes returning invalid JSON. This was addressed in YARN-6082. > However, the fix only fixes the calculation in one site and does not > guarantee to avoid the problem.Likewise, generating NaN/Inf can break the web > GUI if the columns cannot render non-numeric values. > The suggested fix is to check for NaN/Inf in the protob. The protob replaces > NaN/Inf by 0.0f. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9543) UI2 should handle missing ATSv2 gracefully
[ https://issues.apache.org/jira/browse/YARN-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842399#comment-16842399 ] Giovanni Matteo Fumarola commented on YARN-9543: Thanks [~zsiegl]. Do you have any screenshot of before and after your patch? > UI2 should handle missing ATSv2 gracefully > -- > > Key: YARN-9543 > URL: https://issues.apache.org/jira/browse/YARN-9543 > Project: Hadoop YARN > Issue Type: Improvement > Components: ATSv2, yarn-ui-v2 >Affects Versions: 3.1.2 >Reporter: Zoltan Siegl >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-9543.001.patch > > > Resource manager UI2 is throwing some console errors and a error page on > flows page. > Suggested improvements: > * Disable or remove flows tab if ATSv2 is not available/installed > * Handle all connection errors to ATSv2 gracefully -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9505) Add container allocation latency for Opportunistic Scheduler
[ https://issues.apache.org/jira/browse/YARN-9505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842398#comment-16842398 ] Giovanni Matteo Fumarola commented on YARN-9505: The patch looks good. Committed to trunk. Thanks [~abmodi] for working on this and [~elgoiri] for the review. > Add container allocation latency for Opportunistic Scheduler > > > Key: YARN-9505 > URL: https://issues.apache.org/jira/browse/YARN-9505 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9505.001.patch, YARN-9505.002.patch, > YARN-9505.003.patch, YARN-9505.004.patch, YARN-9505.005.patch, > YARN-9505.006.patch, YARN-9505.007.patch > > > This will help in tuning the opportunistic scheduler and it's configuration > parameters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9505) Add container allocation latency for Opportunistic Scheduler
[ https://issues.apache.org/jira/browse/YARN-9505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9505: --- Fix Version/s: 3.3.0 > Add container allocation latency for Opportunistic Scheduler > > > Key: YARN-9505 > URL: https://issues.apache.org/jira/browse/YARN-9505 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9505.001.patch, YARN-9505.002.patch, > YARN-9505.003.patch, YARN-9505.004.patch, YARN-9505.005.patch, > YARN-9505.006.patch, YARN-9505.007.patch > > > This will help in tuning the opportunistic scheduler and it's configuration > parameters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9559) Create AbstractContainersLauncher for pluggable ContainersLauncher logic
[ https://issues.apache.org/jira/browse/YARN-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841774#comment-16841774 ] Giovanni Matteo Fumarola commented on YARN-9559: Thanks [~jhung] for the patch. Can you remove the extra new lines that are not needed? Can you rerun the failed tests TestFederationInterceptor and TestContainerSchedulerQueuing in your local machine? > Create AbstractContainersLauncher for pluggable ContainersLauncher logic > > > Key: YARN-9559 > URL: https://issues.apache.org/jira/browse/YARN-9559 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9559.001.patch, YARN-9559.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9559) Create AbstractContainersLauncher for pluggable ContainersLauncher logic
[ https://issues.apache.org/jira/browse/YARN-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841612#comment-16841612 ] Giovanni Matteo Fumarola commented on YARN-9559: Thanks [~erwaman] for the review but this is a -1 from me. [~jhung] please fix the failed unit tests and the checkstyle warnings. > Create AbstractContainersLauncher for pluggable ContainersLauncher logic > > > Key: YARN-9559 > URL: https://issues.apache.org/jira/browse/YARN-9559 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9559.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9552) FairScheduler: NODE_UPDATE can cause NoSuchElementException
[
https://issues.apache.org/jira/browse/YARN-9552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840663#comment-16840663
]
Giovanni Matteo Fumarola commented on YARN-9552:
Committed [^YARN-9552-004.patch] to trunk. The patch looks good.
Thanks [~pbacsko] for working on this and [~snemeth] for the initial review.
> FairScheduler: NODE_UPDATE can cause NoSuchElementException
> ---
>
> Key: YARN-9552
> URL: https://issues.apache.org/jira/browse/YARN-9552
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9552-001.patch, YARN-9552-002.patch,
> YARN-9552-003.patch, YARN-9552-004.patch
>
>
> We observed a race condition inside YARN with the following stack trace:
> {noformat}
> 18/11/07 06:45:09.559 SchedulerEventDispatcher:Event Processor ERROR
> EventDispatcher: Error in handling event type NODE_UPDATE to the Event
> Dispatcher
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036)
> at
> java.util.concurrent.ConcurrentSkipListSet.first(ConcurrentSkipListSet.java:396)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.getNextPendingAsk(AppSchedulingInfo.java:373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.isOverAMShareLimit(FSAppAttempt.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.assignContainer(FSAppAttempt.java:1373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:353)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:204)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1094)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1183)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:132)
> at
> org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> This is basically the same as the one described in YARN-7382, but the root
> cause is different.
> When we create an application attempt, we create an {{FSAppAttempt}} object.
> This contains an {{AppSchedulingInfo}} which contains a set of
> {{SchedulerRequestKey}}. Initially, this set is empty and only initialized a
> bit later on a separate thread during a state transition:
> {noformat}
> 2019-05-07 15:58:02,659 INFO [RM StateStore dispatcher]
> recovery.RMStateStore (RMStateStore.java:transition(239)) - Storing info for
> app: application_1557237478804_0001
> 2019-05-07 15:58:02,684 INFO [RM Event dispatcher] rmapp.RMAppImpl
> (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change
> from NEW_SAVING to SUBMITTED on event = APP_NEW_SAVED
> 2019-05-07 15:58:02,690 INFO [SchedulerEventDispatcher:Event Processor]
> fair.FairScheduler (FairScheduler.java:addApplication(490)) - Accepted
> application application_1557237478804_0001 from user: bacskop, in queue:
> root.bacskop, currently num of applications: 1
> 2019-05-07 15:58:02,698 INFO [RM Event dispatcher] rmapp.RMAppImpl
> (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change
> from SUBMITTED to ACCEPTED on event = APP_ACCEPTED
> 2019-05-07 15:58:02,731 INFO [RM Event dispatcher]
> resourcemanager.ApplicationMasterService
> (ApplicationMasterService.java:registerAppAttempt(434)) - Registering app
> attempt : appattempt_1557237478804_0001_01
> 2019-05-07 15:58:02,732 INFO [RM Event dispatcher] attempt.RMAppAttemptImpl
> (RMAppAttemptImpl.java:handle(920)) - appattempt_1557237478804_0001_01
> State change from NEW to SUBMITTED on event = START
> 2019-05-07 15:58:02,746 INFO [SchedulerEventDispatcher:Event Processor]
> scheduler.SchedulerApplicationAttempt
> (SchedulerApplicationAttempt.java:(207)) - *** In the constructor of
> SchedulerApplicationAttempt
> 2019-05-07 15:58:02,747 INFO [SchedulerEventDispatcher:Event Processor]
> scheduler.SchedulerApplicationAttempt
> (SchedulerApplicationAttempt.java:(230)) - *** Contents of
> appSchedulingInfo: []
> 2019-05-07 15:58:02,752 INFO [SchedulerEventDispatcher:Event Processor]
>
[jira] [Updated] (YARN-9552) FairScheduler: NODE_UPDATE can cause NoSuchElementException
[
https://issues.apache.org/jira/browse/YARN-9552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9552:
---
Fix Version/s: 3.3.0
> FairScheduler: NODE_UPDATE can cause NoSuchElementException
> ---
>
> Key: YARN-9552
> URL: https://issues.apache.org/jira/browse/YARN-9552
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9552-001.patch, YARN-9552-002.patch,
> YARN-9552-003.patch, YARN-9552-004.patch
>
>
> We observed a race condition inside YARN with the following stack trace:
> {noformat}
> 18/11/07 06:45:09.559 SchedulerEventDispatcher:Event Processor ERROR
> EventDispatcher: Error in handling event type NODE_UPDATE to the Event
> Dispatcher
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036)
> at
> java.util.concurrent.ConcurrentSkipListSet.first(ConcurrentSkipListSet.java:396)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.getNextPendingAsk(AppSchedulingInfo.java:373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.isOverAMShareLimit(FSAppAttempt.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.assignContainer(FSAppAttempt.java:1373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:353)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:204)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1094)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:961)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1183)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:132)
> at
> org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> This is basically the same as the one described in YARN-7382, but the root
> cause is different.
> When we create an application attempt, we create an {{FSAppAttempt}} object.
> This contains an {{AppSchedulingInfo}} which contains a set of
> {{SchedulerRequestKey}}. Initially, this set is empty and only initialized a
> bit later on a separate thread during a state transition:
> {noformat}
> 2019-05-07 15:58:02,659 INFO [RM StateStore dispatcher]
> recovery.RMStateStore (RMStateStore.java:transition(239)) - Storing info for
> app: application_1557237478804_0001
> 2019-05-07 15:58:02,684 INFO [RM Event dispatcher] rmapp.RMAppImpl
> (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change
> from NEW_SAVING to SUBMITTED on event = APP_NEW_SAVED
> 2019-05-07 15:58:02,690 INFO [SchedulerEventDispatcher:Event Processor]
> fair.FairScheduler (FairScheduler.java:addApplication(490)) - Accepted
> application application_1557237478804_0001 from user: bacskop, in queue:
> root.bacskop, currently num of applications: 1
> 2019-05-07 15:58:02,698 INFO [RM Event dispatcher] rmapp.RMAppImpl
> (RMAppImpl.java:handle(903)) - application_1557237478804_0001 State change
> from SUBMITTED to ACCEPTED on event = APP_ACCEPTED
> 2019-05-07 15:58:02,731 INFO [RM Event dispatcher]
> resourcemanager.ApplicationMasterService
> (ApplicationMasterService.java:registerAppAttempt(434)) - Registering app
> attempt : appattempt_1557237478804_0001_01
> 2019-05-07 15:58:02,732 INFO [RM Event dispatcher] attempt.RMAppAttemptImpl
> (RMAppAttemptImpl.java:handle(920)) - appattempt_1557237478804_0001_01
> State change from NEW to SUBMITTED on event = START
> 2019-05-07 15:58:02,746 INFO [SchedulerEventDispatcher:Event Processor]
> scheduler.SchedulerApplicationAttempt
> (SchedulerApplicationAttempt.java:(207)) - *** In the constructor of
> SchedulerApplicationAttempt
> 2019-05-07 15:58:02,747 INFO [SchedulerEventDispatcher:Event Processor]
> scheduler.SchedulerApplicationAttempt
> (SchedulerApplicationAttempt.java:(230)) - *** Contents of
> appSchedulingInfo: []
> 2019-05-07 15:58:02,752 INFO [SchedulerEventDispatcher:Event Processor]
> fair.FairScheduler (FairScheduler.java:addApplicationAttempt(546)) - Added
> Application Attempt appattempt_1557237478804_0001_01 to scheduler from
> user: bacskop
>
[jira] [Updated] (YARN-9453) Clean up code long if-else chain in ApplicationCLI#run
[ https://issues.apache.org/jira/browse/YARN-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9453: --- Fix Version/s: 3.3.0 > Clean up code long if-else chain in ApplicationCLI#run > -- > > Key: YARN-9453 > URL: https://issues.apache.org/jira/browse/YARN-9453 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Wanqiang Ji >Priority: Major > Labels: newbie > Fix For: 3.3.0 > > Attachments: YARN-9453.001.patch, YARN-9453.002.patch, > YARN-9453.003.patch, YARN-9453.004.patch > > > org.apache.hadoop.yarn.client.cli.ApplicationCLI#run is 630 lines long, > contains a long if-else chain and many many conditions. > As a start, the bodies of the conditions could be extracted to methods and a > more clean solution could be introduced to parse the argument values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9453) Clean up code long if-else chain in ApplicationCLI#run
[ https://issues.apache.org/jira/browse/YARN-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838772#comment-16838772 ] Giovanni Matteo Fumarola commented on YARN-9453: Thanks [~jiwq] for the patch. I double checked with the help of a compare tool if you have missed something. [^YARN-9453.004.patch] committed to trunk. Thanks [~snemeth] for the help in reviewing. > Clean up code long if-else chain in ApplicationCLI#run > -- > > Key: YARN-9453 > URL: https://issues.apache.org/jira/browse/YARN-9453 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Wanqiang Ji >Priority: Major > Labels: newbie > Attachments: YARN-9453.001.patch, YARN-9453.002.patch, > YARN-9453.003.patch, YARN-9453.004.patch > > > org.apache.hadoop.yarn.client.cli.ApplicationCLI#run is 630 lines long, > contains a long if-else chain and many many conditions. > As a start, the bodies of the conditions could be extracted to methods and a > more clean solution could be introduced to parse the argument values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9493) Scheduler Page does not display the right page by query string
[ https://issues.apache.org/jira/browse/YARN-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9493: --- Fix Version/s: 3.3.0 > Scheduler Page does not display the right page by query string > -- > > Key: YARN-9493 > URL: https://issues.apache.org/jira/browse/YARN-9493 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager, webapp >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Fix For: 3.3.0 > > Attachments: Actual-1.png, Actual-2.png, YARN-9493.001.patch, > YARN-9493.002.patch, YARN-9493.003.patch > > > In RM when we used the Capacity Scheduler, I found some mistakes caused the > WebApp's scheduler page cannot display the right page by query string. > Some opts that can be reproduced such as: > * Directed by url like [http://rm:8088/cluster/scheduler?openQueues=Queue: > default|http://127.0.0.1:8088/cluster/scheduler?openQueues=Queue:%20default#Queue:%20root] > * Directed by url like > [http://rm:8088/cluster/scheduler?openQueues=Queue:%20default|http://127.0.0.1:8088/cluster/scheduler?openQueues=Queue:%20default#Queue:%20root] > !Actual-1.png! > Except that I found if we repeat click one child queue, the window location > display the error url. > !Actual-2.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9493) Scheduler Page does not display the right page by query string
[ https://issues.apache.org/jira/browse/YARN-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9493: --- Summary: Scheduler Page does not display the right page by query string (was: Fix Scheduler Page can't display the right page by query string) > Scheduler Page does not display the right page by query string > -- > > Key: YARN-9493 > URL: https://issues.apache.org/jira/browse/YARN-9493 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager, webapp >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Attachments: Actual-1.png, Actual-2.png, YARN-9493.001.patch, > YARN-9493.002.patch, YARN-9493.003.patch > > > In RM when we used the Capacity Scheduler, I found some mistakes caused the > WebApp's scheduler page cannot display the right page by query string. > Some opts that can be reproduced such as: > * Directed by url like [http://rm:8088/cluster/scheduler?openQueues=Queue: > default|http://127.0.0.1:8088/cluster/scheduler?openQueues=Queue:%20default#Queue:%20root] > * Directed by url like > [http://rm:8088/cluster/scheduler?openQueues=Queue:%20default|http://127.0.0.1:8088/cluster/scheduler?openQueues=Queue:%20default#Queue:%20root] > !Actual-1.png! > Except that I found if we repeat click one child queue, the window location > display the error url. > !Actual-2.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9453) Clean up code long if-else chain in ApplicationCLI#run
[ https://issues.apache.org/jira/browse/YARN-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837514#comment-16837514 ] Giovanni Matteo Fumarola commented on YARN-9453: Thanks [~snemeth]. I would like to have a new run of Yetus. > Clean up code long if-else chain in ApplicationCLI#run > -- > > Key: YARN-9453 > URL: https://issues.apache.org/jira/browse/YARN-9453 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Wanqiang Ji >Priority: Major > Labels: newbie > Attachments: YARN-9453.001.patch, YARN-9453.002.patch, > YARN-9453.003.patch > > > org.apache.hadoop.yarn.client.cli.ApplicationCLI#run is 630 lines long, > contains a long if-else chain and many many conditions. > As a start, the bodies of the conditions could be extracted to methods and a > more clean solution could be introduced to parse the argument values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9453) Clean up code long if-else chain in ApplicationCLI#run
[ https://issues.apache.org/jira/browse/YARN-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837496#comment-16837496 ] Giovanni Matteo Fumarola commented on YARN-9453: Hi [~jiwq] please rebase the patch to master. > Clean up code long if-else chain in ApplicationCLI#run > -- > > Key: YARN-9453 > URL: https://issues.apache.org/jira/browse/YARN-9453 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Wanqiang Ji >Priority: Major > Labels: newbie > Attachments: YARN-9453.001.patch, YARN-9453.002.patch, > YARN-9453.003.patch > > > org.apache.hadoop.yarn.client.cli.ApplicationCLI#run is 630 lines long, > contains a long if-else chain and many many conditions. > As a start, the bodies of the conditions could be extracted to methods and a > more clean solution could be introduced to parse the argument values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9493) Fix Scheduler Page can't display the right page by query string
[ https://issues.apache.org/jira/browse/YARN-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837495#comment-16837495 ] Giovanni Matteo Fumarola commented on YARN-9493: Hi [~jiwq], can you push a new patch to trigger again the Yetus? The changes looks fine to me. > Fix Scheduler Page can't display the right page by query string > --- > > Key: YARN-9493 > URL: https://issues.apache.org/jira/browse/YARN-9493 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager, webapp >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Attachments: Actual-1.png, Actual-2.png, YARN-9493.001.patch, > YARN-9493.002.patch > > > In RM when we used the Capacity Scheduler, I found some mistakes caused the > WebApp's scheduler page cannot display the right page by query string. > Some opts that can be reproduced such as: > * Directed by url like [http://rm:8088/cluster/scheduler?openQueues=Queue: > default|http://127.0.0.1:8088/cluster/scheduler?openQueues=Queue:%20default#Queue:%20root] > * Directed by url like > [http://rm:8088/cluster/scheduler?openQueues=Queue:%20default|http://127.0.0.1:8088/cluster/scheduler?openQueues=Queue:%20default#Queue:%20root] > !Actual-1.png! > Except that I found if we repeat click one child queue, the window location > display the error url. > !Actual-2.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9522) AppBlock ignores full qualified class name of PseudoAuthenticationHandler
[
https://issues.apache.org/jira/browse/YARN-9522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9522:
---
Fix Version/s: 3.3.0
> AppBlock ignores full qualified class name of PseudoAuthenticationHandler
> -
>
> Key: YARN-9522
> URL: https://issues.apache.org/jira/browse/YARN-9522
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9522-001.patch, YARN-9522-002.patch
>
>
> {{AuthenticationHandler}} can be either configured using fqcn or type.
> {{AppBlock}} checks for only the type simple and ignores the fqcn of
> {{PseudoAuthenticationHandler}} while checking whether ui is secured or not.
> {code}
>* @param authHandler The short-name (or fully qualified class name) of the
>* authentication handler.
> {code}
> *AppBlock.java*
> {code}
> // check if UI is unsecured.
> String httpAuth =
> conf.get(CommonConfigurationKeys.HADOOP_HTTP_AUTHENTICATION_TYPE);
> this.unsecuredUI = (httpAuth != null) && httpAuth.equals("simple");
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9522) AppBlock ignores full qualified class name of PseudoAuthenticationHandler
[
https://issues.apache.org/jira/browse/YARN-9522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836701#comment-16836701
]
Giovanni Matteo Fumarola commented on YARN-9522:
My bad. [^YARN-9522-001.patch] was ok.
Committed v1 to trunk.
Thanks [~Prabhu Joseph].
> AppBlock ignores full qualified class name of PseudoAuthenticationHandler
> -
>
> Key: YARN-9522
> URL: https://issues.apache.org/jira/browse/YARN-9522
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9522-001.patch, YARN-9522-002.patch
>
>
> {{AuthenticationHandler}} can be either configured using fqcn or type.
> {{AppBlock}} checks for only the type simple and ignores the fqcn of
> {{PseudoAuthenticationHandler}} while checking whether ui is secured or not.
> {code}
>* @param authHandler The short-name (or fully qualified class name) of the
>* authentication handler.
> {code}
> *AppBlock.java*
> {code}
> // check if UI is unsecured.
> String httpAuth =
> conf.get(CommonConfigurationKeys.HADOOP_HTTP_AUTHENTICATION_TYPE);
> this.unsecuredUI = (httpAuth != null) && httpAuth.equals("simple");
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9522) AppBlock ignores full qualified class name of PseudoAuthenticationHandler
[
https://issues.apache.org/jira/browse/YARN-9522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836595#comment-16836595
]
Giovanni Matteo Fumarola commented on YARN-9522:
Thanks [~Prabhu Joseph].
Can you add an additional set of parenthesis to make the statements more
readable?
e.g. ( (c1) && (c2)) || c3;
> AppBlock ignores full qualified class name of PseudoAuthenticationHandler
> -
>
> Key: YARN-9522
> URL: https://issues.apache.org/jira/browse/YARN-9522
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9522-001.patch
>
>
> {{AuthenticationHandler}} can be either configured using fqcn or type.
> {{AppBlock}} checks for only the type simple and ignores the fqcn of
> {{PseudoAuthenticationHandler}} while checking whether ui is secured or not.
> {code}
>* @param authHandler The short-name (or fully qualified class name) of the
>* authentication handler.
> {code}
> *AppBlock.java*
> {code}
> // check if UI is unsecured.
> String httpAuth =
> conf.get(CommonConfigurationKeys.HADOOP_HTTP_AUTHENTICATION_TYPE);
> this.unsecuredUI = (httpAuth != null) && httpAuth.equals("simple");
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9483) DistributedShell does not release container when failed to localize at launch
[
https://issues.apache.org/jira/browse/YARN-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9483:
---
Fix Version/s: 3.3.0
> DistributedShell does not release container when failed to localize at launch
> -
>
> Key: YARN-9483
> URL: https://issues.apache.org/jira/browse/YARN-9483
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9483-001.patch
>
>
> DistributedShell does not release container when failed to localize at
> launch. The launch threads does not increment completed & failed containers
> when failed to localize. And the main thread waits for the containers to
> complete without failing the job.
> {code}
> yarn jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -shell_command ls -shell_args / -jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -localize_files /tmp/prabhu
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9483) DistributedShell does not release container when failed to localize at launch
[
https://issues.apache.org/jira/browse/YARN-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836594#comment-16836594
]
Giovanni Matteo Fumarola commented on YARN-9483:
The patch looks good. Committed to trunk.
Thanks [~Prabhu Joseph] for the patch and [~pbacsko] for the initial review.
> DistributedShell does not release container when failed to localize at launch
> -
>
> Key: YARN-9483
> URL: https://issues.apache.org/jira/browse/YARN-9483
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9483-001.patch
>
>
> DistributedShell does not release container when failed to localize at
> launch. The launch threads does not increment completed & failed containers
> when failed to localize. And the main thread waits for the containers to
> complete without failing the job.
> {code}
> yarn jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -shell_command ls -shell_args / -jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -localize_files /tmp/prabhu
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9513) [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater than 8
[
https://issues.apache.org/jira/browse/YARN-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9513:
---
Attachment: YARN-9513.addendum.patch
> [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater
> than 8
> --
>
> Key: YARN-9513
> URL: https://issues.apache.org/jira/browse/YARN-9513
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9513.001.patch, YARN-9513.002.patch,
> YARN-9513.addendum.patch
>
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8:
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.502
> s<<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
> [ERROR]
> testManyRuns(org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker)
> Time elapsed: 0.206 s <<<
> ERROR!org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantViolationException:
> ReferenceError: "GcCountPS_Scavenge" is not defined in at line number
> 1
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantsChecker.logOrThrow(InvariantsChecker.java:74)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.MetricsInvariantChecker.editSchedule(MetricsInvariantChecker.java:180)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker.testManyRuns(TestMetricsInvariantChecker.java:69)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9513) [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater than 8
[
https://issues.apache.org/jira/browse/YARN-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835138#comment-16835138
]
Giovanni Matteo Fumarola commented on YARN-9513:
Thanks [~eyang]. Addendum attached.
> [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater
> than 8
> --
>
> Key: YARN-9513
> URL: https://issues.apache.org/jira/browse/YARN-9513
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9513.001.patch, YARN-9513.002.patch,
> YARN-9513.addendum.patch
>
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8:
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.502
> s<<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
> [ERROR]
> testManyRuns(org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker)
> Time elapsed: 0.206 s <<<
> ERROR!org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantViolationException:
> ReferenceError: "GcCountPS_Scavenge" is not defined in at line number
> 1
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantsChecker.logOrThrow(InvariantsChecker.java:74)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.MetricsInvariantChecker.editSchedule(MetricsInvariantChecker.java:180)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker.testManyRuns(TestMetricsInvariantChecker.java:69)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9513) [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater than 8
[
https://issues.apache.org/jira/browse/YARN-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834982#comment-16834982
]
Giovanni Matteo Fumarola commented on YARN-9513:
Committed [^YARN-9513.002.patch] to trunk.
Thanks [~adam.antal] for the patch and [~snemeth] for the initial review.
> [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater
> than 8
> --
>
> Key: YARN-9513
> URL: https://issues.apache.org/jira/browse/YARN-9513
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9513.001.patch, YARN-9513.002.patch
>
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8:
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.502
> s<<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
> [ERROR]
> testManyRuns(org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker)
> Time elapsed: 0.206 s <<<
> ERROR!org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantViolationException:
> ReferenceError: "GcCountPS_Scavenge" is not defined in at line number
> 1
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantsChecker.logOrThrow(InvariantsChecker.java:74)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.MetricsInvariantChecker.editSchedule(MetricsInvariantChecker.java:180)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker.testManyRuns(TestMetricsInvariantChecker.java:69)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9513) [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater than 8
[
https://issues.apache.org/jira/browse/YARN-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9513:
---
Summary: [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of
JDK greater than 8 (was: [JDK11] Fix TestMetricsInvariantChecker#testManyRuns
in case of JDK greater than 9)
> [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater
> than 8
> --
>
> Key: YARN-9513
> URL: https://issues.apache.org/jira/browse/YARN-9513
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-9513.001.patch, YARN-9513.002.patch
>
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8:
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.502
> s<<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
> [ERROR]
> testManyRuns(org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker)
> Time elapsed: 0.206 s <<<
> ERROR!org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantViolationException:
> ReferenceError: "GcCountPS_Scavenge" is not defined in at line number
> 1
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantsChecker.logOrThrow(InvariantsChecker.java:74)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.MetricsInvariantChecker.editSchedule(MetricsInvariantChecker.java:180)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker.testManyRuns(TestMetricsInvariantChecker.java:69)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9513) [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater than 8
[
https://issues.apache.org/jira/browse/YARN-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9513:
---
Fix Version/s: 3.3.0
> [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater
> than 8
> --
>
> Key: YARN-9513
> URL: https://issues.apache.org/jira/browse/YARN-9513
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9513.001.patch, YARN-9513.002.patch
>
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8:
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.502
> s<<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
> [ERROR]
> testManyRuns(org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker)
> Time elapsed: 0.206 s <<<
> ERROR!org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantViolationException:
> ReferenceError: "GcCountPS_Scavenge" is not defined in at line number
> 1
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantsChecker.logOrThrow(InvariantsChecker.java:74)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.MetricsInvariantChecker.editSchedule(MetricsInvariantChecker.java:180)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker.testManyRuns(TestMetricsInvariantChecker.java:69)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9513) [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater than 9
[
https://issues.apache.org/jira/browse/YARN-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9513:
---
Summary: [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of
JDK greater than 9 (was: [JDK11] TestMetricsInvariantChecker#testManyRuns
InvariantViolationException: ReferenceError: "GcCountPS_Scavenge" is not
defined in at line number 1)
> [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater
> than 9
> --
>
> Key: YARN-9513
> URL: https://issues.apache.org/jira/browse/YARN-9513
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-9513.001.patch, YARN-9513.002.patch
>
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8:
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.502
> s<<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
> [ERROR]
> testManyRuns(org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker)
> Time elapsed: 0.206 s <<<
> ERROR!org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantViolationException:
> ReferenceError: "GcCountPS_Scavenge" is not defined in at line number
> 1
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantsChecker.logOrThrow(InvariantsChecker.java:74)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.MetricsInvariantChecker.editSchedule(MetricsInvariantChecker.java:180)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker.testManyRuns(TestMetricsInvariantChecker.java:69)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9513) [JDK11] TestMetricsInvariantChecker#testManyRuns InvariantViolationException: ReferenceError: "GcCountPS_Scavenge" is not defined in at line number 1
[
https://issues.apache.org/jira/browse/YARN-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834107#comment-16834107
]
Giovanni Matteo Fumarola commented on YARN-9513:
Thanks [~adam.antal].
Please use Shell.isJavaVersionAtLeast(9) instead of
{code:java}
Integer.parseInt(System.getProperty("java.version").split("\\.")[0]) > 8)
{code}
> [JDK11] TestMetricsInvariantChecker#testManyRuns InvariantViolationException:
> ReferenceError: "GcCountPS_Scavenge" is not defined in at line number 1
>
>
> Key: YARN-9513
> URL: https://issues.apache.org/jira/browse/YARN-9513
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-9513.001.patch
>
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8:
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.502
> s<<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
> [ERROR]
> testManyRuns(org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker)
> Time elapsed: 0.206 s <<<
> ERROR!org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantViolationException:
> ReferenceError: "GcCountPS_Scavenge" is not defined in at line number
> 1
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.InvariantsChecker.logOrThrow(InvariantsChecker.java:74)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.MetricsInvariantChecker.editSchedule(MetricsInvariantChecker.java:180)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker.testManyRuns(TestMetricsInvariantChecker.java:69)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
[ https://issues.apache.org/jira/browse/YARN-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9528: --- Fix Version/s: 3.3.0 > Federation RMs starting up at the same time can give duplicate application IDs > -- > > Key: YARN-9528 > URL: https://issues.apache.org/jira/browse/YARN-9528 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9528.01.patch > > > Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9528) Federation RMs starting up at the same time can give duplicate application IDs
[ https://issues.apache.org/jira/browse/YARN-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832870#comment-16832870 ] Giovanni Matteo Fumarola commented on YARN-9528: Thanks [~youchen]. +1 on [^YARN-9528.01.patch] Committed to trunk. > Federation RMs starting up at the same time can give duplicate application IDs > -- > > Key: YARN-9528 > URL: https://issues.apache.org/jira/browse/YARN-9528 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-9528.01.patch > > > Federation RMs starting up at the same time can give duplicate application IDs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6272) TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently
[ https://issues.apache.org/jira/browse/YARN-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826267#comment-16826267 ] Giovanni Matteo Fumarola commented on YARN-6272: Thanks [~Prabhu Joseph] for the patch. I am not a fan of Sleep instructions in unit tests. Can you explain the fix? > TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently > - > > Key: YARN-6272 > URL: https://issues.apache.org/jira/browse/YARN-6272 > Project: Hadoop YARN > Issue Type: Test > Components: yarn >Affects Versions: 3.0.0-alpha4 >Reporter: Ray Chiang >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-6272-001.patch > > > I'm seeing this unit test fail fairly often in trunk: > testAMRMClientWithContainerResourceChange(org.apache.hadoop.yarn.client.api.impl.TestAMRMClient) > Time elapsed: 5.113 sec <<< FAILURE! > java.lang.AssertionError: expected:<1> but was:<0> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.doContainerResourceChange(TestAMRMClient.java:1087) > at > org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientWithContainerResourceChange(TestAMRMClient.java:963) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9507) Fix NPE if NM fails to init
[ https://issues.apache.org/jira/browse/YARN-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9507: --- Summary: Fix NPE if NM fails to init (was: NPE when u stop NM if NM Init failed) > Fix NPE if NM fails to init > --- > > Key: YARN-9507 > URL: https://issues.apache.org/jira/browse/YARN-9507 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-9507-001.patch > > > 2019-04-24 14:06:44,101 WARN org.apache.hadoop.service.AbstractService: When > stopping the service NodeManager > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:492) > at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) > at > org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54) > at > org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102) > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:947) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1018) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9424) Change getDeclaredMethods to getMethods in FederationClientInterceptor#invokeConcurrent()
[ https://issues.apache.org/jira/browse/YARN-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824768#comment-16824768 ] Giovanni Matteo Fumarola commented on YARN-9424: Committed to trunk. > Change getDeclaredMethods to getMethods in > FederationClientInterceptor#invokeConcurrent() > - > > Key: YARN-9424 > URL: https://issues.apache.org/jira/browse/YARN-9424 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Shen Yinjie >Assignee: Shen Yinjie >Priority: Major > Attachments: YARN-9124_1.patch > > > In YARN-8699, FederationClientInterceptor#invokeConcurrent uses > getDeclaredMethods(), which cannot recongnize some methods in > ApplicationBaseProtocol (ApplicationClientProtocol extend > ApplicationBaseProtocol) . > We have implemented some methods in FederationClientInterceptor, such as > getApplications(), GetQueueUserAclsInfo ...etc, when I run "yarn application > -list" by connecting to yarn router, router will throw exception. > So change getDeclaredMethods() to getMethods(). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9501) TestCapacitySchedulerOvercommit#testReducePreemptAndCancel fails intermittent
[jira] [Commented] (YARN-9501) TestCapacitySchedulerOvercommit#testReducePreemptAndCancel fails intermittent
[
https://issues.apache.org/jira/browse/YARN-9501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824626#comment-16824626
]
Giovanni Matteo Fumarola commented on YARN-9501:
Thanks [~Prabhu Joseph] for the patch and [~elgoiri] for the review.
This is similar to YARN-9491. Committed to trunk.
> TestCapacitySchedulerOvercommit#testReducePreemptAndCancel fails intermittent
> -
>
> Key: YARN-9501
> URL: https://issues.apache.org/jira/browse/YARN-9501
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler, test
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9501-001.patch, YARN-9501-002.patch
>
>
> TestCapacitySchedulerOvercommit#testReducePreemptAndCancel fails intermittent
> {code}
> [ERROR]
> testReducePreemptAndCancel(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerOvercommit)
> Time elapsed: 0.729 s <<< FAILURE!
> java.lang.AssertionError: Expected a preemption message
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:712)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerOvercommit.assertPreemption(TestSchedulerOvercommit.java:616)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerOvercommit.testReducePreemptAndCancel(TestSchedulerOvercommit.java:327)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9504) [UI2] Fair scheduler queue view page is broken
[
https://issues.apache.org/jira/browse/YARN-9504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824622#comment-16824622
]
Giovanni Matteo Fumarola commented on YARN-9504:
[~zsiegl] thanks for the patch.
Can you re-upload the patch? I think the system is recognizing one of the 2
pictures as "the last file uploaded".
> [UI2] Fair scheduler queue view page is broken
> --
>
> Key: YARN-9504
> URL: https://issues.apache.org/jira/browse/YARN-9504
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, yarn-ui-v2
>Affects Versions: 3.2.0, 3.3.0, 3.2.1
>Reporter: Zoltan Siegl
>Assignee: Zoltan Siegl
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
> Attachments: Screenshot 2019-04-23 at 14.52.57.png, Screenshot
> 2019-04-23 at 14.59.35.png, YARN-9504.001.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> UI2 queue page currently displays white screen for Fair Scheduler.
>
> In src/main/webapp/app/components/tree-selector.js:377 (getUsedCapacity) code
> refers to
> queueData.get("partitionMap") which is null for fair scheduler queue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9491) TestApplicationMasterServiceFair>ApplicationMasterServiceTestBase.testUpdateTrackingUrl fails intermittent
[
https://issues.apache.org/jira/browse/YARN-9491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824605#comment-16824605
]
Giovanni Matteo Fumarola commented on YARN-9491:
Thanks [~Prabhu Joseph].
Committed to trunk.
> TestApplicationMasterServiceFair>ApplicationMasterServiceTestBase.testUpdateTrackingUrl
> fails intermittent
> --
>
> Key: YARN-9491
> URL: https://issues.apache.org/jira/browse/YARN-9491
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9491-001.patch
>
>
> TestApplicationMasterServiceFair>ApplicationMasterServiceTestBase.testUpdateTrackingUrl
> fails intermittent.
> {code}
> Error Message
> expected:<[hadoop.apache.org]> but was:<[N/A]>
> Stacktrace
> org.junit.ComparisonFailure: expected:<[hadoop.apache.org]> but was:<[N/A]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterServiceTestBase.testUpdateTrackingUrl(ApplicationMasterServiceTestBase.java:467)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9491) TestApplicationMasterServiceFair>ApplicationMasterServiceTestBase.testUpdateTrackingUrl fails intermittent
[
https://issues.apache.org/jira/browse/YARN-9491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9491:
---
Fix Version/s: 3.3.0
> TestApplicationMasterServiceFair>ApplicationMasterServiceTestBase.testUpdateTrackingUrl
> fails intermittent
> --
>
> Key: YARN-9491
> URL: https://issues.apache.org/jira/browse/YARN-9491
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9491-001.patch
>
>
> TestApplicationMasterServiceFair>ApplicationMasterServiceTestBase.testUpdateTrackingUrl
> fails intermittent.
> {code}
> Error Message
> expected:<[hadoop.apache.org]> but was:<[N/A]>
> Stacktrace
> org.junit.ComparisonFailure: expected:<[hadoop.apache.org]> but was:<[N/A]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterServiceTestBase.testUpdateTrackingUrl(ApplicationMasterServiceTestBase.java:467)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9493) Fix Scheduler Page can't display the right page by query string
[ https://issues.apache.org/jira/browse/YARN-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820349#comment-16820349 ] Giovanni Matteo Fumarola commented on YARN-9493: Thanks [~jiwq] for the patch. The findbug and unit test failures are not related to your patch. Please fix the checkstyle warning by splitting the instruction in 2 lines. > Fix Scheduler Page can't display the right page by query string > --- > > Key: YARN-9493 > URL: https://issues.apache.org/jira/browse/YARN-9493 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager, webapp >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Attachments: Actual-1.png, Actual-2.png, YARN-9493.001.patch > > > In RM when we used the Capacity Scheduler, I found some mistakes caused the > WebApp's scheduler page cannot display the right page by query string. > Some opts that can be reproduced such as: > * Directed by url like [http://rm:8088/cluster/scheduler?openQueues=Queue: > default|http://127.0.0.1:8088/cluster/scheduler?openQueues=Queue:%20default#Queue:%20root] > * Directed by url like > [http://rm:8088/cluster/scheduler?openQueues=Queue:%20default|http://127.0.0.1:8088/cluster/scheduler?openQueues=Queue:%20default#Queue:%20root] > !Actual-1.png! > Except that I found if we repeat click one child queue, the window location > display the error url. > !Actual-2.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9435) Add Opportunistic Scheduler metrics in ResourceManager.
[ https://issues.apache.org/jira/browse/YARN-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9435: --- Fix Version/s: 3.3.0 > Add Opportunistic Scheduler metrics in ResourceManager. > --- > > Key: YARN-9435 > URL: https://issues.apache.org/jira/browse/YARN-9435 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9435.001.patch, YARN-9435.002.patch, > YARN-9435.003.patch, YARN-9435.004.patch > > > Right now there are no metrics available for Opportunistic Scheduler at > ResourceManager. As part of this jira, we will add metrics like number of > allocated opportunistic containers, released opportunistic containers, node > level allocations, rack level allocations etc. for Opportunistic Scheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9435) Add Opportunistic Scheduler metrics in ResourceManager.
[ https://issues.apache.org/jira/browse/YARN-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815713#comment-16815713 ] Giovanni Matteo Fumarola commented on YARN-9435: Thanks [~abmodi] for the patch. Committed to trunk. > Add Opportunistic Scheduler metrics in ResourceManager. > --- > > Key: YARN-9435 > URL: https://issues.apache.org/jira/browse/YARN-9435 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9435.001.patch, YARN-9435.002.patch, > YARN-9435.003.patch, YARN-9435.004.patch > > > Right now there are no metrics available for Opportunistic Scheduler at > ResourceManager. As part of this jira, we will add metrics like number of > allocated opportunistic containers, released opportunistic containers, node > level allocations, rack level allocations etc. for Opportunistic Scheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9462) TestResourceTrackerService.testNodeRemovalGracefully fails sporadically
[
https://issues.apache.org/jira/browse/YARN-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815623#comment-16815623
]
Giovanni Matteo Fumarola commented on YARN-9462:
Thanks [~Prabhu Joseph] for the patch.
I am not a fan of increasing time values to avoid tests to fail because it
hides the real problem.
Please run the test multiple times to verify your fix. e.g with a use of a bash
script, with intelliJ or with Repeat annotation.
> TestResourceTrackerService.testNodeRemovalGracefully fails sporadically
> ---
>
> Key: YARN-9462
> URL: https://issues.apache.org/jira/browse/YARN-9462
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, test
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments:
> TestResourceTrackerService.testNodeRemovalGracefully.txt, YARN-9462-001.patch
>
>
> TestResourceTrackerService.testNodeRemovalGracefully fails sporadically
> {code}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService.testNodeRemovalUtilDecomToUntracked(TestResourceTrackerService.java:2318)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService.testNodeRemovalUtil(TestResourceTrackerService.java:2280)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService.testNodeRemovalGracefully(TestResourceTrackerService.java:2133)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[ https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813826#comment-16813826 ] Giovanni Matteo Fumarola commented on YARN-999: --- Committed the addendum to trunk. Thanks [~ste...@apache.org] for raising the issue and [~elgoiri] for fixing it. > In case of long running tasks, reduce node resource should balloon out > resource quickly by calling preemption API and suspending running task. > --- > > Key: YARN-999 > URL: https://issues.apache.org/jira/browse/YARN-999 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Íñigo Goiri >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-291.000.patch, YARN-999.001.patch, > YARN-999.002.patch, YARN-999.003.patch, YARN-999.004.patch, > YARN-999.005.patch, YARN-999.006.patch, YARN-999.007.patch, > YARN-999.008.patch, YARN-999.009.patch, YARN-999.010.patch, > YARN-999.addendum.patch > > > In current design and implementation, when we decrease resource on node to > less than resource consumption of current running tasks, tasks can still be > running until the end. But just no new task get assigned on this node > (because AvailableResource < 0) until some tasks are finished and > AvailableResource > 0 again. This is good for most cases but in case of long > running task, it could be too slow for resource setting to actually work so > preemption could be used here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[ https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813795#comment-16813795 ] Giovanni Matteo Fumarola commented on YARN-999: --- The addendum looks ok. Should we commit to unblock the branch or just wait yetus result? > In case of long running tasks, reduce node resource should balloon out > resource quickly by calling preemption API and suspending running task. > --- > > Key: YARN-999 > URL: https://issues.apache.org/jira/browse/YARN-999 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Íñigo Goiri >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-291.000.patch, YARN-999.001.patch, > YARN-999.002.patch, YARN-999.003.patch, YARN-999.004.patch, > YARN-999.005.patch, YARN-999.006.patch, YARN-999.007.patch, > YARN-999.008.patch, YARN-999.009.patch, YARN-999.010.patch, > YARN-999.addendum.patch > > > In current design and implementation, when we decrease resource on node to > less than resource consumption of current running tasks, tasks can still be > running until the end. But just no new task get assigned on this node > (because AvailableResource < 0) until some tasks are finished and > AvailableResource > 0 again. This is good for most cases but in case of long > running task, it could be too slow for resource setting to actually work so > preemption could be used here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[ https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813740#comment-16813740 ] Giovanni Matteo Fumarola commented on YARN-999: --- Thanks [~ste...@apache.org], we are fixing it right now. > In case of long running tasks, reduce node resource should balloon out > resource quickly by calling preemption API and suspending running task. > --- > > Key: YARN-999 > URL: https://issues.apache.org/jira/browse/YARN-999 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Íñigo Goiri >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-291.000.patch, YARN-999.001.patch, > YARN-999.002.patch, YARN-999.003.patch, YARN-999.004.patch, > YARN-999.005.patch, YARN-999.006.patch, YARN-999.007.patch, > YARN-999.008.patch, YARN-999.009.patch, YARN-999.010.patch > > > In current design and implementation, when we decrease resource on node to > less than resource consumption of current running tasks, tasks can still be > running until the end. But just no new task get assigned on this node > (because AvailableResource < 0) until some tasks are finished and > AvailableResource > 0 again. This is good for most cases but in case of long > running task, it could be too slow for resource setting to actually work so > preemption could be used here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9435) Add Opportunistic Scheduler metrics in ResourceManager.
[ https://issues.apache.org/jira/browse/YARN-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813679#comment-16813679 ] Giovanni Matteo Fumarola commented on YARN-9435: Thanks [~abmodi] for the patch. Why do we need a Thread.sleep(1000); in the unit test? > Add Opportunistic Scheduler metrics in ResourceManager. > --- > > Key: YARN-9435 > URL: https://issues.apache.org/jira/browse/YARN-9435 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9435.001.patch, YARN-9435.002.patch, > YARN-9435.003.patch > > > Right now there are no metrics available for Opportunistic Scheduler at > ResourceManager. As part of this jira, we will add metrics like number of > allocated opportunistic containers, released opportunistic containers, node > level allocations, rack level allocations etc. for Opportunistic Scheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[ https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813674#comment-16813674 ] Giovanni Matteo Fumarola commented on YARN-999: --- Committed to trunk. > In case of long running tasks, reduce node resource should balloon out > resource quickly by calling preemption API and suspending running task. > --- > > Key: YARN-999 > URL: https://issues.apache.org/jira/browse/YARN-999 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Íñigo Goiri >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-291.000.patch, YARN-999.001.patch, > YARN-999.002.patch, YARN-999.003.patch, YARN-999.004.patch, > YARN-999.005.patch, YARN-999.006.patch, YARN-999.007.patch, > YARN-999.008.patch, YARN-999.009.patch, YARN-999.010.patch > > > In current design and implementation, when we decrease resource on node to > less than resource consumption of current running tasks, tasks can still be > running until the end. But just no new task get assigned on this node > (because AvailableResource < 0) until some tasks are finished and > AvailableResource > 0 again. This is good for most cases but in case of long > running task, it could be too slow for resource setting to actually work so > preemption could be used here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[ https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-999: -- Fix Version/s: 3.3.0 > In case of long running tasks, reduce node resource should balloon out > resource quickly by calling preemption API and suspending running task. > --- > > Key: YARN-999 > URL: https://issues.apache.org/jira/browse/YARN-999 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Íñigo Goiri >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-291.000.patch, YARN-999.001.patch, > YARN-999.002.patch, YARN-999.003.patch, YARN-999.004.patch, > YARN-999.005.patch, YARN-999.006.patch, YARN-999.007.patch, > YARN-999.008.patch, YARN-999.009.patch, YARN-999.010.patch > > > In current design and implementation, when we decrease resource on node to > less than resource consumption of current running tasks, tasks can still be > running until the end. But just no new task get assigned on this node > (because AvailableResource < 0) until some tasks are finished and > AvailableResource > 0 again. This is good for most cases but in case of long > running task, it could be too slow for resource setting to actually work so > preemption could be used here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[ https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812822#comment-16812822 ] Giovanni Matteo Fumarola commented on YARN-999: --- Thanks [~elgoiri] for the hard work. [^YARN-999.010.patch] looks good. I will commit tomorrow if there will be no additional comments. > In case of long running tasks, reduce node resource should balloon out > resource quickly by calling preemption API and suspending running task. > --- > > Key: YARN-999 > URL: https://issues.apache.org/jira/browse/YARN-999 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Íñigo Goiri >Priority: Major > Attachments: YARN-291.000.patch, YARN-999.001.patch, > YARN-999.002.patch, YARN-999.003.patch, YARN-999.004.patch, > YARN-999.005.patch, YARN-999.006.patch, YARN-999.007.patch, > YARN-999.008.patch, YARN-999.009.patch, YARN-999.010.patch > > > In current design and implementation, when we decrease resource on node to > less than resource consumption of current running tasks, tasks can still be > running until the end. But just no new task get assigned on this node > (because AvailableResource < 0) until some tasks are finished and > AvailableResource > 0 again. This is good for most cases but in case of long > running task, it could be too slow for resource setting to actually work so > preemption could be used here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9437) RMNodeImpls occupy too much memory and causes RM GC to take a long time
[ https://issues.apache.org/jira/browse/YARN-9437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9437: --- Priority: Minor (was: Blocker) > RMNodeImpls occupy too much memory and causes RM GC to take a long time > --- > > Key: YARN-9437 > URL: https://issues.apache.org/jira/browse/YARN-9437 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.1 >Reporter: qiuliang >Priority: Minor > Attachments: 1.png, 2.png, 3.png > > > We use hadoop-2.9.1 in our production environment with 1600+ nodes. 95.63% of > RM memory is occupied by RMNodeImpl. Analysis of RM memory found that each > RMNodeImpl has approximately 14M. The reason is that there is a 130,000+ > completedcontainers in each RMNodeImpl that has not been released. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[
https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811171#comment-16811171
]
Giovanni Matteo Fumarola edited comment on YARN-999 at 4/5/19 7:06 PM:
---
Thanks [~elgoiri] for the hard work to add this important feature.
The patch is fully tested with good unit tests.`
Few comments on the last patch:
* Few times you used containers launched. It should be launched containers.
* Line 253 LOG.debug("{} is overcommitted ({}), preempt/kill containers",
should be LOG.info. I would like to search in the code what the situation.
* You can use the word Overcommitted over "over committed".
* In testGetRunningContainersToKill some comments have the numeration missing.
e.g AM instead of AM0.
* Line 1069 false, ExecutionType.GUARANTEED, "GUARANTEED0"); it should
be GUARANTEED1
* I am not a fan of Static imports. when should you use static import? Very
sparingly! Only use it when you'd otherwise be tempted to declare local copies
of constants, or to abuse inheritance (the Constant Interface Antipattern). ...
If you overuse the static import feature, it can make your program unreadable
and unmaintainable, polluting its namespace with all the static members you
import. Readers of your code (including you, a few months after you wrote it)
will not know which class a static member comes from. Importing all of the
static members from a class can be particularly harmful to readability; if you
need only one or two members, import them individually.
[Static import
documentation|https://docs.oracle.com/javase/8/docs/technotes/guides/language/static-import.html)].
I would remove those from the new tests classes.
was (Author: giovanni.fumarola):
Thanks [~elgoiri] for the hard work to add this important feature.
The patch is fully tested with good unit tests.`
Few comments on the last patch:
* Few times you used containers launched. It should be launched containers.
* Line 253 LOG.debug("{} is overcommitted ({}), preempt/kill containers",
should be LOG.info. I would like to search in the code what the situation.
* You can use the word Overcommitted over "over committed".
* In testGetRunningContainersToKill some comments have the numeration missing.
e.g AM instead of AM0.
* Line 1069 false, ExecutionType.GUARANTEED, "GUARANTEED0"); it should
be GUARANTEED1
* I am not a fan of Static imports. when should you use static import? Very
sparingly! Only use it when you'd otherwise be tempted to declare local copies
of constants, or to abuse inheritance (the Constant Interface Antipattern). ...
If you overuse the static import feature, it can make your program unreadable
and unmaintainable, polluting its namespace with all the static members you
import. Readers of your code (including you, a few months after you wrote it)
will not know which class a static member comes from. Importing all of the
static members from a class can be particularly harmful to readability; if you
need only one or two members, import them individually.
([https://docs.oracle.com/javase/8/docs/technotes/guides/language/static-import.html)
]I would remove those from the new tests classes.
> In case of long running tasks, reduce node resource should balloon out
> resource quickly by calling preemption API and suspending running task.
> ---
>
> Key: YARN-999
> URL: https://issues.apache.org/jira/browse/YARN-999
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: graceful, nodemanager, scheduler
>Reporter: Junping Du
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: YARN-291.000.patch, YARN-999.001.patch,
> YARN-999.002.patch, YARN-999.003.patch, YARN-999.004.patch,
> YARN-999.005.patch, YARN-999.006.patch, YARN-999.007.patch,
> YARN-999.008.patch, YARN-999.009.patch
>
>
> In current design and implementation, when we decrease resource on node to
> less than resource consumption of current running tasks, tasks can still be
> running until the end. But just no new task get assigned on this node
> (because AvailableResource < 0) until some tasks are finished and
> AvailableResource > 0 again. This is good for most cases but in case of long
> running task, it could be too slow for resource setting to actually work so
> preemption could be used here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[
https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811171#comment-16811171
]
Giovanni Matteo Fumarola commented on YARN-999:
---
Thanks [~elgoiri] for the hard work to add this important feature.
The patch is fully tested with good unit tests.`
Few comments on the last patch:
* Few times you used containers launched. It should be launched containers.
* Line 253 LOG.debug("{} is overcommitted ({}), preempt/kill containers",
should be LOG.info. I would like to search in the code what the situation.
* You can use the word Overcommitted over "over committed".
* In testGetRunningContainersToKill some comments have the numeration missing.
e.g AM instead of AM0.
* Line 1069 false, ExecutionType.GUARANTEED, "GUARANTEED0"); it should
be GUARANTEED1
* I am not a fan of Static imports. when should you use static import? Very
sparingly! Only use it when you'd otherwise be tempted to declare local copies
of constants, or to abuse inheritance (the Constant Interface Antipattern). ...
If you overuse the static import feature, it can make your program unreadable
and unmaintainable, polluting its namespace with all the static members you
import. Readers of your code (including you, a few months after you wrote it)
will not know which class a static member comes from. Importing all of the
static members from a class can be particularly harmful to readability; if you
need only one or two members, import them individually.
([https://docs.oracle.com/javase/8/docs/technotes/guides/language/static-import.html)
]I would remove those from the new tests classes.
> In case of long running tasks, reduce node resource should balloon out
> resource quickly by calling preemption API and suspending running task.
> ---
>
> Key: YARN-999
> URL: https://issues.apache.org/jira/browse/YARN-999
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: graceful, nodemanager, scheduler
>Reporter: Junping Du
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: YARN-291.000.patch, YARN-999.001.patch,
> YARN-999.002.patch, YARN-999.003.patch, YARN-999.004.patch,
> YARN-999.005.patch, YARN-999.006.patch, YARN-999.007.patch,
> YARN-999.008.patch, YARN-999.009.patch
>
>
> In current design and implementation, when we decrease resource on node to
> less than resource consumption of current running tasks, tasks can still be
> running until the end. But just no new task get assigned on this node
> (because AvailableResource < 0) until some tasks are finished and
> AvailableResource > 0 again. This is good for most cases but in case of long
> running task, it could be too slow for resource setting to actually work so
> preemption could be used here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9428) Add metrics for paused containers in NodeManager
[ https://issues.apache.org/jira/browse/YARN-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9428: --- Fix Version/s: 3.3.0 > Add metrics for paused containers in NodeManager > > > Key: YARN-9428 > URL: https://issues.apache.org/jira/browse/YARN-9428 > Project: Hadoop YARN > Issue Type: Task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9428.001.patch, YARN-9428.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9428) Add metrics for paused containers in NodeManager
[ https://issues.apache.org/jira/browse/YARN-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9428: --- Description: Add metrics for paused containers in NodeManager. > Add metrics for paused containers in NodeManager > > > Key: YARN-9428 > URL: https://issues.apache.org/jira/browse/YARN-9428 > Project: Hadoop YARN > Issue Type: Task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9428.001.patch, YARN-9428.002.patch > > > Add metrics for paused containers in NodeManager. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9431) Fix flaky junit test fair.TestAppRunnability after YARN-8967
[
https://issues.apache.org/jira/browse/YARN-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807058#comment-16807058
]
Giovanni Matteo Fumarola commented on YARN-9431:
Thanks [~wilfreds] for the patch.
LGTM +1.
Committed to trunk.
> Fix flaky junit test fair.TestAppRunnability after YARN-8967
>
>
> Key: YARN-9431
> URL: https://issues.apache.org/jira/browse/YARN-9431
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, test
>Affects Versions: 3.3.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Minor
> Attachments: YARN-9431.001.patch
>
>
> In YARN-4901 one of the scheduler tests failed. This seems to be linked to
> the changes around the placement rules introduced in YARN-8967.
> Applications submitted in the tests are accepted and rejected at the same
> time:
> {code}
> 2019-04-01 12:00:57,269 INFO [main] fair.FairScheduler
> (FairScheduler.java:addApplication(540)) - Accepted application
> application_0_0001 from user: user1, in queue: root.user1, currently num of
> applications: 1
> 2019-04-01 12:00:57,269 INFO [AsyncDispatcher event handler]
> fair.FairScheduler (FairScheduler.java:rejectApplicationWithMessage(1344)) -
> Reject application application_0_0001 submitted by user user1 application
> rejected by placement rules.
> {code}
> This should never happen and is most likely due to the way the tests
> generates the application and events.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9431) Fix flaky junit test fair.TestAppRunnability after YARN-8967
[
https://issues.apache.org/jira/browse/YARN-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9431:
---
Fix Version/s: 3.3.0
> Fix flaky junit test fair.TestAppRunnability after YARN-8967
>
>
> Key: YARN-9431
> URL: https://issues.apache.org/jira/browse/YARN-9431
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, test
>Affects Versions: 3.3.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9431.001.patch
>
>
> In YARN-4901 one of the scheduler tests failed. This seems to be linked to
> the changes around the placement rules introduced in YARN-8967.
> Applications submitted in the tests are accepted and rejected at the same
> time:
> {code}
> 2019-04-01 12:00:57,269 INFO [main] fair.FairScheduler
> (FairScheduler.java:addApplication(540)) - Accepted application
> application_0_0001 from user: user1, in queue: root.user1, currently num of
> applications: 1
> 2019-04-01 12:00:57,269 INFO [AsyncDispatcher event handler]
> fair.FairScheduler (FairScheduler.java:rejectApplicationWithMessage(1344)) -
> Reject application application_0_0001 submitted by user user1 application
> rejected by placement rules.
> {code}
> This should never happen and is most likely due to the way the tests
> generates the application and events.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9431) Fix flaky junit test fair.TestAppRunnability after YARN-8967
[
https://issues.apache.org/jira/browse/YARN-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9431:
---
Summary: Fix flaky junit test fair.TestAppRunnability after YARN-8967
(was: flaky junit test fair.TestAppRunnability after YARN-8967)
> Fix flaky junit test fair.TestAppRunnability after YARN-8967
>
>
> Key: YARN-9431
> URL: https://issues.apache.org/jira/browse/YARN-9431
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, test
>Affects Versions: 3.3.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Minor
> Attachments: YARN-9431.001.patch
>
>
> In YARN-4901 one of the scheduler tests failed. This seems to be linked to
> the changes around the placement rules introduced in YARN-8967.
> Applications submitted in the tests are accepted and rejected at the same
> time:
> {code}
> 2019-04-01 12:00:57,269 INFO [main] fair.FairScheduler
> (FairScheduler.java:addApplication(540)) - Accepted application
> application_0_0001 from user: user1, in queue: root.user1, currently num of
> applications: 1
> 2019-04-01 12:00:57,269 INFO [AsyncDispatcher event handler]
> fair.FairScheduler (FairScheduler.java:rejectApplicationWithMessage(1344)) -
> Reject application application_0_0001 submitted by user user1 application
> rejected by placement rules.
> {code}
> This should never happen and is most likely due to the way the tests
> generates the application and events.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9428) Add metrics for paused containers in NodeManager
[
https://issues.apache.org/jira/browse/YARN-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807050#comment-16807050
]
Giovanni Matteo Fumarola edited comment on YARN-9428 at 4/1/19 6:14 PM:
Thanks [~abmodi] for the change.
A minor change.
Can you change this instruction:
@Metric MutableGaugeInt containersPaused;
with
@Metric("# of paused containers") MutableGaugeInt containersPaused;
was (Author: giovanni.fumarola):
Thanks [~abmodi]
A minor change.
Can you change this instruction:
@Metric MutableGaugeInt containersPaused;
with
@Metric("# of paused containers") MutableGaugeInt containersPaused;
> Add metrics for paused containers in NodeManager
>
>
> Key: YARN-9428
> URL: https://issues.apache.org/jira/browse/YARN-9428
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-9428.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9428) Add metrics for paused containers in NodeManager
[
https://issues.apache.org/jira/browse/YARN-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807050#comment-16807050
]
Giovanni Matteo Fumarola commented on YARN-9428:
Thanks [~abmodi]
A minor change.
Can you change this instruction:
@Metric MutableGaugeInt containersPaused;
with
@Metric("# of paused containers") MutableGaugeInt containersPaused;
> Add metrics for paused containers in NodeManager
>
>
> Key: YARN-9428
> URL: https://issues.apache.org/jira/browse/YARN-9428
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-9428.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9418) ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics
[
https://issues.apache.org/jira/browse/YARN-9418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807045#comment-16807045
]
Giovanni Matteo Fumarola commented on YARN-9418:
Thanks [~Prabhu Joseph] for the patch.
LGTM +1.
Committed to trunk (The title in the commit shows
/apps//entities/YARN_CONTAINER).
> ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics
>
>
> Key: YARN-9418
> URL: https://issues.apache.org/jira/browse/YARN-9418
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: YARN-9418-001.patch, YARN-9418-002.patch,
> YARN-9418-003.patch
>
>
> ATSV2 entities rest api does not show the metrics
> {code:java}
> [hbase@yarn-ats-3 centos]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553685341603_0006/entities/YARN_CONTAINER/container_e18_1553685341603_0006_01_01?user.name=hbase=METRICS;
> | jq .
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553695002014,
> "idprefix": 0,
> "type": "YARN_CONTAINER",
> "id": "container_e18_1553685341603_0006_01_01",
> "info": {
> "UID":
> "ats!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01",
> "FROM_ID":
> "ats!hbase!QuasiMonteCarlo!1553695001394!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{code}
> NodeManager puts YARN_CONTAINER entities with CPU and MEMORY metrics but this
> is not shown in above output. Found NM container entities are set with
> entityIdPrefix as inverted container starttime whereas RM container entities
> are set with default 0. TimelineReader fetches only RM container entries.
> Confirmed with setting NM container entities entityIdPrefix to 0 same as RM
> (for testing purpose) and found metrics are shown.
> {code:java}
> "metrics": [
> {
> "type": "SINGLE_VALUE",
> "id": "MEMORY",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 490430464
> }
> },
> {
> "type": "SINGLE_VALUE",
> "id": "CPU",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 5
> }
> }
> ]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9418) ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics
[
https://issues.apache.org/jira/browse/YARN-9418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9418:
---
Fix Version/s: 3.3.0
> ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics
>
>
> Key: YARN-9418
> URL: https://issues.apache.org/jira/browse/YARN-9418
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: YARN-9418-001.patch, YARN-9418-002.patch,
> YARN-9418-003.patch
>
>
> ATSV2 entities rest api does not show the metrics
> {code:java}
> [hbase@yarn-ats-3 centos]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553685341603_0006/entities/YARN_CONTAINER/container_e18_1553685341603_0006_01_01?user.name=hbase=METRICS;
> | jq .
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553695002014,
> "idprefix": 0,
> "type": "YARN_CONTAINER",
> "id": "container_e18_1553685341603_0006_01_01",
> "info": {
> "UID":
> "ats!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01",
> "FROM_ID":
> "ats!hbase!QuasiMonteCarlo!1553695001394!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{code}
> NodeManager puts YARN_CONTAINER entities with CPU and MEMORY metrics but this
> is not shown in above output. Found NM container entities are set with
> entityIdPrefix as inverted container starttime whereas RM container entities
> are set with default 0. TimelineReader fetches only RM container entries.
> Confirmed with setting NM container entities entityIdPrefix to 0 same as RM
> (for testing purpose) and found metrics are shown.
> {code:java}
> "metrics": [
> {
> "type": "SINGLE_VALUE",
> "id": "MEMORY",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 490430464
> }
> },
> {
> "type": "SINGLE_VALUE",
> "id": "CPU",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 5
> }
> }
> ]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9418) ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics
[
https://issues.apache.org/jira/browse/YARN-9418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9418:
---
Summary: ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show
metrics (was: ATSV2 /apps/${appId}/entities/YARN_CONTAINER rest api does not
show metrics)
> ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics
>
>
> Key: YARN-9418
> URL: https://issues.apache.org/jira/browse/YARN-9418
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Attachments: YARN-9418-001.patch, YARN-9418-002.patch,
> YARN-9418-003.patch
>
>
> ATSV2 entities rest api does not show the metrics
> {code:java}
> [hbase@yarn-ats-3 centos]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553685341603_0006/entities/YARN_CONTAINER/container_e18_1553685341603_0006_01_01?user.name=hbase=METRICS;
> | jq .
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553695002014,
> "idprefix": 0,
> "type": "YARN_CONTAINER",
> "id": "container_e18_1553685341603_0006_01_01",
> "info": {
> "UID":
> "ats!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01",
> "FROM_ID":
> "ats!hbase!QuasiMonteCarlo!1553695001394!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{code}
> NodeManager puts YARN_CONTAINER entities with CPU and MEMORY metrics but this
> is not shown in above output. Found NM container entities are set with
> entityIdPrefix as inverted container starttime whereas RM container entities
> are set with default 0. TimelineReader fetches only RM container entries.
> Confirmed with setting NM container entities entityIdPrefix to 0 same as RM
> (for testing purpose) and found metrics are shown.
> {code:java}
> "metrics": [
> {
> "type": "SINGLE_VALUE",
> "id": "MEMORY",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 490430464
> }
> },
> {
> "type": "SINGLE_VALUE",
> "id": "CPU",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 5
> }
> }
> ]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9227) DistributedShell RelativePath is not removed at end
[
https://issues.apache.org/jira/browse/YARN-9227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807033#comment-16807033
]
Giovanni Matteo Fumarola commented on YARN-9227:
Thanks [~Prabhu Joseph] for the patch and [~snemeth] for the review. Committed
to trunk.
> DistributedShell RelativePath is not removed at end
> ---
>
> Key: YARN-9227
> URL: https://issues.apache.org/jira/browse/YARN-9227
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-shell
>Affects Versions: 3.1.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: 0001-YARN-9227.patch, 0002-YARN-9227.patch,
> 0003-YARN-9227.patch, YARN-9227-004.patch, YARN-9227-005.patch
>
>
> DistributedShell Job does not remove the relative path which contains jars
> and localized files.
> {code}
> [ambari-qa@ash hadoop-yarn]$ hadoop fs -ls
> /user/ambari-qa/DistributedShell/application_1542665708563_0017
> Found 2 items
> -rw-r--r-- 3 ambari-qa hdfs 46636 2019-01-23 13:37
> /user/ambari-qa/DistributedShell/application_1542665708563_0017/AppMaster.jar
> -rwx--x--- 3 ambari-qa hdfs 4 2019-01-23 13:37
> /user/ambari-qa/DistributedShell/application_1542665708563_0017/shellCommands
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9227) DistributedShell RelativePath is not removed at end
[
https://issues.apache.org/jira/browse/YARN-9227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated YARN-9227:
---
Fix Version/s: 3.3.0
> DistributedShell RelativePath is not removed at end
> ---
>
> Key: YARN-9227
> URL: https://issues.apache.org/jira/browse/YARN-9227
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-shell
>Affects Versions: 3.1.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: 0001-YARN-9227.patch, 0002-YARN-9227.patch,
> 0003-YARN-9227.patch, YARN-9227-004.patch, YARN-9227-005.patch
>
>
> DistributedShell Job does not remove the relative path which contains jars
> and localized files.
> {code}
> [ambari-qa@ash hadoop-yarn]$ hadoop fs -ls
> /user/ambari-qa/DistributedShell/application_1542665708563_0017
> Found 2 items
> -rw-r--r-- 3 ambari-qa hdfs 46636 2019-01-23 13:37
> /user/ambari-qa/DistributedShell/application_1542665708563_0017/AppMaster.jar
> -rwx--x--- 3 ambari-qa hdfs 4 2019-01-23 13:37
> /user/ambari-qa/DistributedShell/application_1542665708563_0017/shellCommands
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9424) Change getDeclaredMethods to getMethods in FederationClientInterceptor#invokeConcurrent()
[
https://issues.apache.org/jira/browse/YARN-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805287#comment-16805287
]
Giovanni Matteo Fumarola commented on YARN-9424:
Thanks [~shenyinjie] for the patch. From online:
{{"getDeclaredMethods}} includes all methods declared _by the class itself_,
whereas {{getMethods}} returns only public methods, but also those inherited
from a base class."
We need this change since it includes other methods.
+1.
> Change getDeclaredMethods to getMethods in
> FederationClientInterceptor#invokeConcurrent()
> -
>
> Key: YARN-9424
> URL: https://issues.apache.org/jira/browse/YARN-9424
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Shen Yinjie
>Assignee: Shen Yinjie
>Priority: Major
> Attachments: YARN-9124_1.patch
>
>
> In YARN-8699, FederationClientInterceptor#invokeConcurrent uses
> getDeclaredMethods(), which cannot recongnize some methods in
> ApplicationBaseProtocol (ApplicationClientProtocol extend
> ApplicationBaseProtocol) .
> We have implemented some methods in FederationClientInterceptor, such as
> getApplications(), GetQueueUserAclsInfo ...etc, when I run "yarn application
> -list" by connecting to yarn router, router will throw exception.
> So change getDeclaredMethods() to getMethods().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
