[jira] [Created] (YARN-10793) Upgrade Junit from 4 to 5 in hadoop-yarn-server-applicationhistoryservice
ANANDA G B created YARN-10793: - Summary: Upgrade Junit from 4 to 5 in hadoop-yarn-server-applicationhistoryservice Key: YARN-10793 URL: https://issues.apache.org/jira/browse/YARN-10793 Project: Hadoop YARN Issue Type: Sub-task Reporter: ANANDA G B Upgrade Junit from 4 to 5 in hadoop-yarn-server-applicationhistoryservice -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10543) Timeline Server V1.5 not supporting audit log
[ https://issues.apache.org/jira/browse/YARN-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348105#comment-17348105 ] ANANDA G B commented on YARN-10543: --- [~zhuqi] Thanks for your review. I will write UT and attach patch today. > Timeline Server V1.5 not supporting audit log > - > > Key: YARN-10543 > URL: https://issues.apache.org/jira/browse/YARN-10543 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: TimeLine > Attachments: YARN-10543-001.patch, YARN-10543-002.patch > > > Like JHS, TS V1.5 can also support audit log when Timeline REST APIs are > accessed. This will helps to know the operation performed on TS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10543) Timeline Server V1.5 not supporting audit log
[ https://issues.apache.org/jira/browse/YARN-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10543: -- Attachment: YARN-10543-002.patch > Timeline Server V1.5 not supporting audit log > - > > Key: YARN-10543 > URL: https://issues.apache.org/jira/browse/YARN-10543 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: TimeLine > Attachments: YARN-10543-001.patch, YARN-10543-002.patch > > > Like JHS, TS V1.5 can also support audit log when Timeline REST APIs are > accessed. This will helps to know the operation performed on TS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10543) Timeline Server V1.5 not supporting audit log
[ https://issues.apache.org/jira/browse/YARN-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10543: -- Attachment: YARN-10543-001.patch > Timeline Server V1.5 not supporting audit log > - > > Key: YARN-10543 > URL: https://issues.apache.org/jira/browse/YARN-10543 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: TimeLine > Attachments: YARN-10543-001.patch > > > Like JHS, TS V1.5 can also support audit log when Timeline REST APIs are > accessed. This will helps to know the operation performed on TS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10543) Timeline Server V1.5 not supporting audit log
[ https://issues.apache.org/jira/browse/YARN-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10543: -- Attachment: (was: YARN-10543-001.patch) > Timeline Server V1.5 not supporting audit log > - > > Key: YARN-10543 > URL: https://issues.apache.org/jira/browse/YARN-10543 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: TimeLine > > Like JHS, TS V1.5 can also support audit log when Timeline REST APIs are > accessed. This will helps to know the operation performed on TS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340154#comment-17340154 ] ANANDA G B edited comment on YARN-10258 at 5/6/21, 1:49 PM: [~ebadger], [~BilwaST], [~brahmareddy] Fixed the UT and updated the patch, can you review and merge to trunk was (Author: gb.ana...@gmail.com): [~BilwaST] [~brahmareddy] [~Hemanth Boyina] Fixed the UT and updated the patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258-005.patch, YARN-10258-006.patch, > YARN-10258-007.patch, YARN-10258-008.patch, YARN-10258-009.patch, > YARN-10258-010.patch, YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340154#comment-17340154 ] ANANDA G B commented on YARN-10258: --- [~BilwaST] [~brahmareddy] [~Hemanth Boyina] Fixed the UT and updated the patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258-005.patch, YARN-10258-006.patch, > YARN-10258-007.patch, YARN-10258-008.patch, YARN-10258-009.patch, > YARN-10258-010.patch, YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340131#comment-17340131 ] ANANDA G B commented on YARN-10258: --- Updated latest patch with UT fix > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258-005.patch, YARN-10258-006.patch, > YARN-10258-007.patch, YARN-10258-008.patch, YARN-10258-009.patch, > YARN-10258-010.patch, YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10258: -- Attachment: YARN-10258-010.patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258-005.patch, YARN-10258-006.patch, > YARN-10258-007.patch, YARN-10258-008.patch, YARN-10258-009.patch, > YARN-10258-010.patch, YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10258: -- Attachment: YARN-10258-009.patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258-005.patch, YARN-10258-006.patch, > YARN-10258-007.patch, YARN-10258-008.patch, YARN-10258-009.patch, > YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297827#comment-17297827
]
ANANDA G B edited comment on YARN-9731 at 3/9/21, 9:09 AM:
---
Thanks for working on it [~magnum].
[~prabhujoseph] Can we commit this 3.1.2 and 3.3.1 version.
was (Author: gb.ana...@gmail.com):
Thanks for working on it [~magnum].
[~prabhujoseph] Can we commit this 3.1.2 version.
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> YARN-9731.003.patch, YARN-9731.004.patch, YARN-9731.005.patch,
> ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297827#comment-17297827
]
ANANDA G B commented on YARN-9731:
--
Thanks for working on it [~magnum].
[~prabhujoseph] Can we commit this 3.1.2 version.
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> YARN-9731.003.patch, YARN-9731.004.patch, YARN-9731.005.patch,
> ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10669) Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN on RM switch and TS restart
[
https://issues.apache.org/jira/browse/YARN-10669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ANANDA G B reassigned YARN-10669:
-
Assignee: ANANDA G B
> Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN on RM switch and TS
> restart
> --
>
> Key: YARN-10669
> URL: https://issues.apache.org/jira/browse/YARN-10669
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.1.1
> Environment: 3 Nodes Hadoop Secure cluster with 3.1.1 version
>Reporter: Sushanta Sen
>Assignee: ANANDA G B
>Priority: Major
>
> Using delegation token rather than the keytab of the user when submitting job
> to yarn.
> And this config yarn.timeline-service.enabled = true.
> So addTimelineDelegationToken will be executed. My Job has submitted
> successfully, but the question is my job failed when I Switched RM and TS
> restart because TIMELINE_DELEGATION_TOKEN renew failed.
> Only RM switch and TS restart will reproduce the issue.
> RM log snippet below:
> {noformat}
> 2020-12-02 17:37:21,268 | WARN | DelegationTokenRenewer #3402 | Unable to
> add the application to the delegation token renewer. |
> DelegationTokenRenewer.java:949
> java.io.IOException: Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN,
> Service: 192.168.0.2:8190, Ident: (TIMELINE_DELEGATION_TOKEN owner=bnn,
> renewer=mapred, realUser=executor, issueDate=1606880472758,
> maxDate=1607485272758, sequenceNumber=11581, masterKeyId=13)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:508)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$1100(DelegationTokenRenewer.java:80)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:945)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:922)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: HTTP status [403], message
> [org.apache.hadoop.security.token.SecretManager$InvalidToken: Unable to find
> master key for keyId=13 from cache. Failed to renew an unexpired token
> (TIMELINE_DELEGATION_TOKEN owner=bnn, renewer=mapred, realUser=executor,
> issueDate=1606880472758, maxDate=1607485272758, sequenceNumber=11581,
> masterKeyId=13) with sequenceNumber=11581]
> at
> org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:174)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:323)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:239)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:426)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:247)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:227)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineConnector$TimelineClientRetryOpForOperateDelegationToken.run(TimelineConnector.java:431)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineConnector$TimelineClientConnectionRetry.retryOn(TimelineConnector.java:334)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineConnector.operateDelegationToken(TimelineConnector.java:218)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.renewDelegationToken(TimelineClientImpl.java:250)
> at
> org.apache.hadoop.yarn.security.client.TimelineDelegationTokenIdentifier$Renewer.renew(TimelineDelegationTokenIdentifier.java:81)
> at org.apache.hadoop.security.token.Token.renew(Token.java:490)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:634)
> at
>
[jira] [Assigned] (YARN-10666) In ProcfsBasedProcessTree reading smaps file show Permission denied
[ https://issues.apache.org/jira/browse/YARN-10666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B reassigned YARN-10666: - Assignee: ANANDA G B > In ProcfsBasedProcessTree reading smaps file show Permission denied > --- > > Key: YARN-10666 > URL: https://issues.apache.org/jira/browse/YARN-10666 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sushanta Sen >Assignee: ANANDA G B >Priority: Major > > When job submitter user is other than NM's user. > Then NM failed to read /proc//smaps file. > Because smaps file is owned by job submitter user, which is not able to read > by NM's user. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10629) Add a timeout when the AM container is not allocated resources within the timeout period then kill the application.
[ https://issues.apache.org/jira/browse/YARN-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10629: -- Attachment: YARN-10629-002.patch > Add a timeout when the AM container is not allocated resources within the > timeout period then kill the application. > --- > > Key: YARN-10629 > URL: https://issues.apache.org/jira/browse/YARN-10629 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Labels: resourcemanager > Fix For: 3.1.1 > > Attachments: YARN-10629-001.patch, YARN-10629-002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10543) Timeline Server V1.5 not supporting audit log
[ https://issues.apache.org/jira/browse/YARN-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10543: -- Attachment: YARN-10543-001.patch > Timeline Server V1.5 not supporting audit log > - > > Key: YARN-10543 > URL: https://issues.apache.org/jira/browse/YARN-10543 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: TimeLine > Fix For: 3.1.1 > > Attachments: YARN-10543-001.patch > > > Like JHS, TS V1.5 can also support audit log when Timeline REST APIs are > accessed. This will helps to know the operation performed on TS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10258: -- Attachment: YARN-10258-008.patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258-005.patch, YARN-10258-006.patch, > YARN-10258-007.patch, YARN-10258-008.patch, YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10258: -- Attachment: YARN-10258-007.patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258-005.patch, YARN-10258-006.patch, > YARN-10258-007.patch, YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10258: -- Attachment: YARN-10258-006.patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258-005.patch, YARN-10258-006.patch, > YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10629) Add a timeout when the AM container is not allocated resources within the timeout period then kill the application.
[ https://issues.apache.org/jira/browse/YARN-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10629: -- Attachment: YARN-10629-001.patch > Add a timeout when the AM container is not allocated resources within the > timeout period then kill the application. > --- > > Key: YARN-10629 > URL: https://issues.apache.org/jira/browse/YARN-10629 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Labels: resourcemanager > Fix For: 3.1.1 > > Attachments: YARN-10629-001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10258: -- Attachment: YARN-10258-005.patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258-005.patch, YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10258: -- Attachment: YARN-10258_004.patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10544) AMParams.java having un-necessary access identifier static final
[ https://issues.apache.org/jira/browse/YARN-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10544: -- Attachment: YARN-10544_001.patch > AMParams.java having un-necessary access identifier static final > > > Key: YARN-10544 > URL: https://issues.apache.org/jira/browse/YARN-10544 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Trivial > Attachments: YARN-10544_001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10258: -- Attachment: YARN-10258-003.patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288411#comment-17288411 ] ANANDA G B commented on YARN-10258: --- Thanks [~BilwaST] and [~zhuqi] for review. Resubmitting path to trigger build. > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10629) Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/YARN-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10629: -- Labels: resourcemanager (was: ipv6) > Avoid unsafe split and append on fields that might be IPv6 literals > --- > > Key: YARN-10629 > URL: https://issues.apache.org/jira/browse/YARN-10629 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Labels: resourcemanager > Fix For: 3.1.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10629) Add a timeout when the AM container is not allocated resources within the timeout period then kill the application.
[ https://issues.apache.org/jira/browse/YARN-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10629: -- Summary: Add a timeout when the AM container is not allocated resources within the timeout period then kill the application. (was: Avoid unsafe split and append on fields that might be IPv6 literals) > Add a timeout when the AM container is not allocated resources within the > timeout period then kill the application. > --- > > Key: YARN-10629 > URL: https://issues.apache.org/jira/browse/YARN-10629 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Labels: resourcemanager > Fix For: 3.1.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10258: -- Attachment: YARN-10258-001.patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Fix For: 3.1.3 > > Attachments: YARN-10258-001.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10629) Avoid unsafe split and append on fields that might be IPv6 literals
ANANDA G B created YARN-10629: - Summary: Avoid unsafe split and append on fields that might be IPv6 literals Key: YARN-10629 URL: https://issues.apache.org/jira/browse/YARN-10629 Project: Hadoop YARN Issue Type: Improvement Components: yarn Affects Versions: 3.1.1 Reporter: ANANDA G B Assignee: ANANDA G B Fix For: 3.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10544) AMParams.java having un-necessary access identifier static final
[ https://issues.apache.org/jira/browse/YARN-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B reassigned YARN-10544: - Assignee: ANANDA G B > AMParams.java having un-necessary access identifier static final > > > Key: YARN-10544 > URL: https://issues.apache.org/jira/browse/YARN-10544 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10544) AMParams.java having un-necessary access identifier static final
ANANDA G B created YARN-10544: - Summary: AMParams.java having un-necessary access identifier static final Key: YARN-10544 URL: https://issues.apache.org/jira/browse/YARN-10544 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Reporter: ANANDA G B -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10543) Timeline Server V1.5 not supporting audit log
[ https://issues.apache.org/jira/browse/YARN-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B reassigned YARN-10543: - Assignee: ANANDA G B > Timeline Server V1.5 not supporting audit log > - > > Key: YARN-10543 > URL: https://issues.apache.org/jira/browse/YARN-10543 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: TimeLine > Fix For: 3.1.1 > > > Like JHS, TS V1.5 can also support audit log when Timeline REST APIs are > accessed. This will helps to know the operation performed on TS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10543) Timeline Server V1.5 not supporting audit log
ANANDA G B created YARN-10543: - Summary: Timeline Server V1.5 not supporting audit log Key: YARN-10543 URL: https://issues.apache.org/jira/browse/YARN-10543 Project: Hadoop YARN Issue Type: Improvement Components: timelineserver Affects Versions: 3.1.1 Reporter: ANANDA G B Fix For: 3.1.1 Like JHS, TS V1.5 can also support audit log when Timeline REST APIs are accessed. This will helps to know the operation performed on TS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
ANANDA G B created YARN-10258: - Summary: Add metrics for 'ApplicationsRunning' in NodeManager Key: YARN-10258 URL: https://issues.apache.org/jira/browse/YARN-10258 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 3.1.3 Reporter: ANANDA G B Assignee: ANANDA G B Fix For: 3.1.3 Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904839#comment-16904839 ] ANANDA G B commented on YARN-9681: -- [~eepayne]: Can you review my analysis and fix. > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.1, 3.1.2 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: patch > Attachments: After running job on queue1.png, Before running job on > queue1.png, YARN-9681.0001.patch, YARN-9681.0002.patch, YARN-9681.0003.patch, > YARN-9681.0004.patch, YARN-9681.0005.patch > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9720) MR job submitted to a queue with default partition accessing the non-exclusive label resources
[ https://issues.apache.org/jira/browse/YARN-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904838#comment-16904838 ] ANANDA G B commented on YARN-9720: -- [~eepayne]: can you check this, i have added capacity-scheduler.xml > MR job submitted to a queue with default partition accessing the > non-exclusive label resources > -- > > Key: YARN-9720 > URL: https://issues.apache.org/jira/browse/YARN-9720 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler, resourcemanager >Affects Versions: 3.1.1, 3.1.2 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Attachments: Issue.png > > > When MR job is submitted to a queue1 with default partition, then it is > accessing non-exclusive partition resources. Please find the attachments. > MR Job command: > ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.0201.jar > pi -Dmapreduce.job.queuename=queue1 -Dmapreduce.job.node-label-expression= 10 > 10 > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-9681: - Attachment: YARN-9681.0005.patch > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.1, 3.1.2 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: patch > Attachments: After running job on queue1.png, Before running job on > queue1.png, YARN-9681.0001.patch, YARN-9681.0002.patch, YARN-9681.0003.patch, > YARN-9681.0004.patch, YARN-9681.0005.patch > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9681) AM resource limit is incorrect for queue
[
https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903442#comment-16903442
]
ANANDA G B commented on YARN-9681:
--
The problem we have seen is: assignContainers() invokes
LeafQueue.updateCurrentResourceLimits method, where:
*Line1:* this.*{color:#59afe1}cachedResourceLimitsForHeadroom{color}* =
new ResourceLimits(currentResourceLimits.getLimit());
*Line2:* Resource queueMaxResource = getEffectiveMaxCapacityDown(
RMNodeLabelsManager.{color:#59afe1}*NO_LABEL*{color}, minimumAllocation);
*Line3:* this.{color:#59afe1}*cachedResourceLimitsForHeadroom*{color}
.setLimit(Resources._min_(resourceCalculator, clusterResource,
queueMaxResource, currentResourceLimits.getLimit()));
*In Line1:* Setting the *{color:#59afe1}cachedResourceLimitsForHeadroom{color}*
{color:#33}with {color}currentResourceLimits
Which is pool1's queue1 resource limit.
*In Line2:* Setting the queueMaxResource
Which is DEFAULT_PARTITION's queue1 resource limit.
*In Line3:* Setting *{color:#59afe1}cachedResourceLimitsForHeadroom{color}*
{color:#33}= Minimum of queueMaxResource and currentResourceLimits{color}
{color:#33}Which is pool1's queue1 resource limit. (Look at the attached
images for partitions and queues information){color}
{color:#33}So finally,
*{color:#59afe1}cachedResourceLimitsForHeadroom{color}* {color}is setted with
the pool1's queue1 resource limit. Then, same
*{color:#59afe1}cachedResourceLimitsForHeadroom{color}* value is used to
calculate MAXIMUM AM RESOURCE of both the partitions (DEFAULT_PARTITION's and
pool1's). So MAXIMUM AM resource is calculated wrongly for DEFAULT_PARTITION's.
*So, solution is {color:#59afe1}cachedResourceLimitsForHeadroom{color}* must be
maintained for each partitions. So it can be a map where key is partition name
and value is the resource limit.
> AM resource limit is incorrect for queue
>
>
> Key: YARN-9681
> URL: https://issues.apache.org/jira/browse/YARN-9681
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.1.1, 3.1.2
>Reporter: ANANDA G B
>Assignee: ANANDA G B
>Priority: Major
> Labels: patch
> Attachments: After running job on queue1.png, Before running job on
> queue1.png, YARN-9681.0001.patch, YARN-9681.0002.patch, YARN-9681.0003.patch,
> YARN-9681.0004.patch
>
>
> After running the job on Queue1 of Partition1, then Queue1 of
> DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly.
> Please find the attachement.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-9681) AM resource limit is incorrect for queue
[
https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ANANDA G B updated YARN-9681:
-
Comment: was deleted
(was: The problem we have seen is: assignContainers() invokes
LeafQueue.updateCurrentResourceLimits method, where:
*Line1:* this.*{color:#59afe1}cachedResourceLimitsForHeadroom{color}* =
new ResourceLimits(currentResourceLimits.getLimit());
*Line2:* Resource queueMaxResource = getEffectiveMaxCapacityDown(
RMNodeLabelsManager.{color:#59afe1}*NO_LABEL*{color}, minimumAllocation);
*Line3:* this.{color:#59afe1}*cachedResourceLimitsForHeadroom*{color}
.setLimit(Resources._min_(resourceCalculator, clusterResource,
queueMaxResource, currentResourceLimits.getLimit()));
*In Line1:* Setting the *{color:#59afe1}cachedResourceLimitsForHeadroom{color}*
{color:#33}with {color}currentResourceLimits
Which is pool1's queue1 resource limit.
*In Line2:* Setting the queueMaxResource
Which is DEFAULT_PARTITION's queue1 resource limit.
*In Line3:* Setting *{color:#59afe1}cachedResourceLimitsForHeadroom{color}*
{color:#33}= Minimum of queueMaxResource and currentResourceLimits{color}
{color:#33}Which is pool1's queue1 resource limit. (Look at the attached
images for partitions and queues information){color}
{color:#33}So finally,
*{color:#59afe1}cachedResourceLimitsForHeadroom{color}* is setted with the
resource limits of pool1's queue1. Then, this
*{color:#59afe1}cachedResourceLimitsForHeadroom{color}* value is used to
calculate MAX AM RESOURCE of both the partitions (DEFAULT_PARTITION's and
pool1's). So MAX AM resource is calculated wrongly for DEFAULT_PARTITION's.
{color}
{color:#33}So, solution is
*{color:#59afe1}cachedResourceLimitsForHeadroom{color}* must be maintained for
each partitions. So it can be a map where key is partition and value is the
resource limit.{color}
)
> AM resource limit is incorrect for queue
>
>
> Key: YARN-9681
> URL: https://issues.apache.org/jira/browse/YARN-9681
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.1.1, 3.1.2
>Reporter: ANANDA G B
>Assignee: ANANDA G B
>Priority: Major
> Labels: patch
> Attachments: After running job on queue1.png, Before running job on
> queue1.png, YARN-9681.0001.patch, YARN-9681.0002.patch, YARN-9681.0003.patch,
> YARN-9681.0004.patch
>
>
> After running the job on Queue1 of Partition1, then Queue1 of
> DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly.
> Please find the attachement.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9681) AM resource limit is incorrect for queue
[
https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903438#comment-16903438
]
ANANDA G B commented on YARN-9681:
--
The problem we have seen is: assignContainers() invokes
LeafQueue.updateCurrentResourceLimits method, where:
*Line1:* this.*{color:#59afe1}cachedResourceLimitsForHeadroom{color}* =
new ResourceLimits(currentResourceLimits.getLimit());
*Line2:* Resource queueMaxResource = getEffectiveMaxCapacityDown(
RMNodeLabelsManager.{color:#59afe1}*NO_LABEL*{color}, minimumAllocation);
*Line3:* this.{color:#59afe1}*cachedResourceLimitsForHeadroom*{color}
.setLimit(Resources._min_(resourceCalculator, clusterResource,
queueMaxResource, currentResourceLimits.getLimit()));
*In Line1:* Setting the *{color:#59afe1}cachedResourceLimitsForHeadroom{color}*
{color:#33}with {color}currentResourceLimits
Which is pool1's queue1 resource limit.
*In Line2:* Setting the queueMaxResource
Which is DEFAULT_PARTITION's queue1 resource limit.
*In Line3:* Setting *{color:#59afe1}cachedResourceLimitsForHeadroom{color}*
{color:#33}= Minimum of queueMaxResource and currentResourceLimits{color}
{color:#33}Which is pool1's queue1 resource limit. (Look at the attached
images for partitions and queues information){color}
{color:#33}So finally,
*{color:#59afe1}cachedResourceLimitsForHeadroom{color}* is setted with the
resource limits of pool1's queue1. Then, this
*{color:#59afe1}cachedResourceLimitsForHeadroom{color}* value is used to
calculate MAX AM RESOURCE of both the partitions (DEFAULT_PARTITION's and
pool1's). So MAX AM resource is calculated wrongly for DEFAULT_PARTITION's.
{color}
{color:#33}So, solution is
*{color:#59afe1}cachedResourceLimitsForHeadroom{color}* must be maintained for
each partitions. So it can be a map where key is partition and value is the
resource limit.{color}
> AM resource limit is incorrect for queue
>
>
> Key: YARN-9681
> URL: https://issues.apache.org/jira/browse/YARN-9681
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.1.1, 3.1.2
>Reporter: ANANDA G B
>Assignee: ANANDA G B
>Priority: Major
> Labels: patch
> Attachments: After running job on queue1.png, Before running job on
> queue1.png, YARN-9681.0001.patch, YARN-9681.0002.patch, YARN-9681.0003.patch,
> YARN-9681.0004.patch
>
>
> After running the job on Queue1 of Partition1, then Queue1 of
> DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly.
> Please find the attachement.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903174#comment-16903174 ] ANANDA G B commented on YARN-9681: -- Hi Eric Payne, In YARN 5788. *Actual* Am limit in scheduler UI is still based on old resource. *Expected* AM limit to be updated based new partition resource. This is fine. In current Jira: *Actual:* _*After running the job*_ on queue1 of pool1, then queue1 of DEFAULT_PARTITIONs AM resource limit is setting based on the *effective capacity* of queue1 of _*pool1*_. *Expected* Even after running the job on queue1 of pool1, then queue1 of DEFAULT_PARTITIONs AM resource limit must be set based on the *effective capacity* of queue1 of _*DEFAULT_PARTITION*s_. > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.1, 3.1.2 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: patch > Attachments: After running job on queue1.png, Before running job on > queue1.png, YARN-9681.0001.patch, YARN-9681.0002.patch, YARN-9681.0003.patch, > YARN-9681.0004.patch > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9720) MR job submitted to a queue with default partition accessing the non-exclusive label resources
[ https://issues.apache.org/jira/browse/YARN-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903167#comment-16903167 ] ANANDA G B commented on YARN-9720: -- Hi Eric Payne, here is my CapacityScheduler.xml configuration: yarn.scheduler.capacity.maximum-applications 1 Maximum number of applications that can be pending and running. yarn.scheduler.capacity.resource-calculator org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator The ResourceCalculator implementation to be used to compare Resources in the scheduler. The default i.e. DefaultResourceCalculator only uses Memory while DominantResourceCalculator uses dominant-resource to compare multi-dimensional resources such as Memory, CPU etc. yarn.scheduler.capacity.root.queues default,root-default,queue1 The queues at the this level (root is the root queue). yarn.scheduler.capacity.root.accessible-node-labels pool1 yarn.scheduler.capacity.root.accessible-node-labels.pool1.capacity 100 yarn.scheduler.capacity.root.maximum-am-resource-percent 1 yarn.scheduler.capacity.root.default.capacity 20 yarn.scheduler.capacity.root.default.maximum-capacity 100 yarn.scheduler.capacity.root.default.state RUNNING yarn.scheduler.capacity.root.default.maximum-am-resource-percent 0.1 yarn.scheduler.capacity.root.default.accessible-node-labels yarn.scheduler.capacity.root.root-default.capacity 70.0 yarn.scheduler.capacity.root.root-default.maximum-capacity 100 yarn.scheduler.capacity.root.root-default.state RUNNING yarn.scheduler.capacity.root.root-default.maximum-am-resource-percent 0.1 yarn.scheduler.capacity.root.root-default.accessible-node-labels pool1 yarn.scheduler.capacity.root.root-default.default-node-label-expression pool1 yarn.scheduler.capacity.root.root-default.accessible-node-labels.pool1.capacity 80.0 yarn.scheduler.capacity.root.root-default.accessible-node-labels.pool1.maximum-capacity 100.0 yarn.scheduler.capacity.root.queue1.capacity 10.0 yarn.scheduler.capacity.root.queue1.maximum-capacity 100 yarn.scheduler.capacity.root.queue1.state RUNNING yarn.scheduler.capacity.root.queue1.maximum-am-resource-percent 0.8 yarn.scheduler.capacity.root.queue1.accessible-node-labels pool1 yarn.scheduler.capacity.root.queue1.default-node-label-expression pool1 yarn.scheduler.capacity.root.queue1.accessible-node-labels.pool1.capacity 20.0 yarn.scheduler.capacity.root.queue1.accessible-node-labels.pool1.maximum-capacity 100.0 yarn.scheduler.capacity.root.default.user-limit-factor 1 Default queue user limit a percentage from 0.0 to 1.0. yarn.scheduler.capacity.root.default.acl_submit_applications * The ACL of who can submit jobs to the default queue. yarn.scheduler.capacity.root.default.acl_administer_queue * The ACL of who can administer jobs on the default queue. yarn.scheduler.capacity.root.default.acl_application_max_priority * The ACL of who can submit applications with configured priority. yarn.scheduler.capacity.root.default.maximum-application-lifetime -1 Maximum lifetime of an application which is submitted to a queue in seconds. Any value less than or equal to zero will be considered as disabled. This will be a hard time limit for all applications in this queue. If positive value is configured then any application submitted to this queue will be killed after exceeds the configured lifetime. User can also specify lifetime per application basis in application submission context. But user lifetime will be overridden if it exceeds queue maximum lifetime. It is point-in-time configuration. Note : Configuring too low value will result in killing application sooner. This feature is applicable only for leaf queue. yarn.scheduler.capacity.root.default.default-application-lifetime -1 Default lifetime of an application which is submitted to a queue in seconds. Any value less than or equal to zero will be considered as disabled. If the user has not submitted application with lifetime value then this value will be taken. It is point-in-time configuration. Note : Default lifetime can't exceed maximum lifetime. This feature is applicable only for leaf queue. yarn.scheduler.capacity.node-locality-delay 40 Number of missed scheduling opportunities after which the CapacityScheduler attempts to schedule rack-local containers. When setting this parameter, the size of the cluster should be taken into account. We use 40 as the default value, which is approximately the number of nodes in one rack. Note, if this value is -1, the locality constraint in the container request will be ignored, which disables the delay scheduling. yarn.scheduler.capacity.rack-locality-additional-delay -1 Number of additional missed scheduling opportunities over the node-locality-delay ones, after which the
[jira] [Created] (YARN-9720) MR job submitted to a queue with default partition accessing the non-exclusive label resources
ANANDA G B created YARN-9720: Summary: MR job submitted to a queue with default partition accessing the non-exclusive label resources Key: YARN-9720 URL: https://issues.apache.org/jira/browse/YARN-9720 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler, resourcemanager Affects Versions: 3.1.2, 3.1.1 Reporter: ANANDA G B Assignee: ANANDA G B Attachments: Issue.png When MR job is submitted to a queue1 with default partition, then it is accessing non-exclusive partition resources. Please find the attachments. MR Job command: ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.0201.jar pi -Dmapreduce.job.queuename=queue1 -Dmapreduce.job.node-label-expression= 10 10 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-9681: - Attachment: YARN-9681.0004.patch > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.1, 3.1.2 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: patch > Attachments: After running job on queue1.png, Before running job on > queue1.png, YARN-9681.0001.patch, YARN-9681.0002.patch, YARN-9681.0003.patch, > YARN-9681.0004.patch > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895778#comment-16895778 ] ANANDA G B commented on YARN-9681: -- Hi [~sunilg] [~leftnoteasy] [~bibinchundatt] Can you please review the code. > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.1, 3.1.2 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: patch > Attachments: After running job on queue1.png, Before running job on > queue1.png, YARN-9681.0001.patch, YARN-9681.0002.patch, YARN-9681.0003.patch > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-9681: - Attachment: YARN-9681.0003.patch > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.1, 3.1.2 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: patch > Attachments: After running job on queue1.png, Before running job on > queue1.png, YARN-9681.0001.patch, YARN-9681.0002.patch, YARN-9681.0003.patch > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-9681: - Attachment: YARN-9681.0002.patch > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.1, 3.1.2 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: patch > Attachments: After running job on queue1.png, Before running job on > queue1.png, YARN-9681.0001.patch, YARN-9681.0002.patch > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-9681: - Attachment: (was: YARN-9681..patch) > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.1, 3.1.2 >Reporter: ANANDA G B >Priority: Major > Labels: patch > Attachments: After running job on queue1.png, Before running job on > queue1.png, YARN-9681.0001.patch > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-9681: - Attachment: YARN-9681.0001.patch > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.1, 3.1.2 >Reporter: ANANDA G B >Priority: Major > Labels: patch > Attachments: After running job on queue1.png, Before running job on > queue1.png, YARN-9681..patch, YARN-9681.0001.patch > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-9681: - Fix Version/s: (was: 3.1.2) > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.1, 3.1.2 >Reporter: ANANDA G B >Priority: Major > Labels: patch > Attachments: After running job on queue1.png, Before running job on > queue1.png, YARN-9681..patch > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891998#comment-16891998 ] ANANDA G B commented on YARN-9681: -- Hi, [~sunilg], [~bibinchundatt], [~leftnoteasy] I have attached the patch can you please review it. > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.1, 3.1.2 >Reporter: ANANDA G B >Priority: Major > Labels: patch > Fix For: 3.1.2 > > Attachments: After running job on queue1.png, Before running job on > queue1.png, YARN-9681..patch > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887085#comment-16887085 ] ANANDA G B commented on YARN-9681: -- Hi Sunil When we submit a app to queue1 in pool1 (Please check the attached screenshots for nodelabels configuration), the LeafQueue.assignContainers function will update cachedResourceLimitsForHeadroom to nodelabel pool1's resource 8196. Then LeafQueue.activateApplications function calculate the 'Maximum Application Master Resources' of DEFAULT_PARTITION using cachedResourceLimitsForHeadroom. This is wrong actually. So, each nodelabel should have its own cachedResourceLimitsForHeadroom. So it can be a ConcurrentHashMap where key is nodelabel and value is ResourceLimits. > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.2 >Reporter: ANANDA G B >Priority: Major > Attachments: After running job on queue1.png, Before running job on > queue1.png > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-9681: - Comment: was deleted (was: When we submit a app to queue1 in pool1 (Please check the attached screenshots for nodelabels configuration), the LeafQueue.assignContainers function will update cachedResourceLimitsForHeadroom to nodelabel pool1's resource 8196. Then LeafQueue.activateApplications function calculate the 'Maximum Application Master Resources' of DEFAULT_PARTITION using cachedResourceLimitsForHeadroom. This is wrong actually. So, each nodelabel should have its own cachedResourceLimitsForHeadroom. So it can be a ConcurrentHashMap where key is nodelabel and value is ResourceLimits.) > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.2 >Reporter: ANANDA G B >Priority: Major > Attachments: After running job on queue1.png, Before running job on > queue1.png > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887084#comment-16887084 ] ANANDA G B commented on YARN-9681: -- When we submit a app to queue1 in pool1 (Please check the attached screenshots for nodelabels configuration), the LeafQueue.assignContainers function will update cachedResourceLimitsForHeadroom to nodelabel pool1's resource 8196. Then LeafQueue.activateApplications function calculate the 'Maximum Application Master Resources' of DEFAULT_PARTITION using cachedResourceLimitsForHeadroom. This is wrong actually. So, each nodelabel should have its own cachedResourceLimitsForHeadroom. So it can be a ConcurrentHashMap where key is nodelabel and value is ResourceLimits. > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.2 >Reporter: ANANDA G B >Priority: Major > Attachments: After running job on queue1.png, Before running job on > queue1.png > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-9681: - Comment: was deleted (was: I have partition like below: !image-2019-07-17-19-43-16-718.png! When we submit a app to queue1 in pool1, the LeafQueue.assignContainers function will update cachedResourceLimitsForHeadroom to nodelabel pool1's resource 8196. Then LeafQueue.activateApplications function calculate the 'Maximum Application Master Resources' of DEFAULT_PARTITION using cachedResourceLimitsForHeadroom, which is wrong. So, Each nodelabel should have its own cachedResourceLimitsForHeadroom. So it can be a ConcurrentHashMap where key is nodelabel and value is ResourceLimits.) > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.2 >Reporter: ANANDA G B >Priority: Major > Attachments: After running job on queue1.png, Before running job on > queue1.png > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9681) AM resource limit is incorrect for queue
[ https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887083#comment-16887083 ] ANANDA G B commented on YARN-9681: -- I have partition like below: !image-2019-07-17-19-43-16-718.png! When we submit a app to queue1 in pool1, the LeafQueue.assignContainers function will update cachedResourceLimitsForHeadroom to nodelabel pool1's resource 8196. Then LeafQueue.activateApplications function calculate the 'Maximum Application Master Resources' of DEFAULT_PARTITION using cachedResourceLimitsForHeadroom, which is wrong. So, Each nodelabel should have its own cachedResourceLimitsForHeadroom. So it can be a ConcurrentHashMap where key is nodelabel and value is ResourceLimits. > AM resource limit is incorrect for queue > > > Key: YARN-9681 > URL: https://issues.apache.org/jira/browse/YARN-9681 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.2 >Reporter: ANANDA G B >Priority: Major > Attachments: After running job on queue1.png, Before running job on > queue1.png > > > After running the job on Queue1 of Partition1, then Queue1 of > DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. > Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9681) AM resource limit is incorrect for queue
ANANDA G B created YARN-9681: Summary: AM resource limit is incorrect for queue Key: YARN-9681 URL: https://issues.apache.org/jira/browse/YARN-9681 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.1.2 Reporter: ANANDA G B Attachments: After running job on queue1.png, Before running job on queue1.png After running the job on Queue1 of Partition1, then Queue1 of DEFAULT_PARTITION's 'Max Application Master Resources' is calculated wrongly. Please find the attachement. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9674) Max AM Resource calculation is wrong
[ https://issues.apache.org/jira/browse/YARN-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884880#comment-16884880 ] ANANDA G B commented on YARN-9674: -- @[~sunilg] Can you please check this @[~sunilg] One more issue is, after running the job on particular queue (Say Queue1 of partition1), its resetting the Max AM Resources of Default Partitions Queue1. So can i raise separate Jira for it ? > Max AM Resource calculation is wrong > > > Key: YARN-9674 > URL: https://issues.apache.org/jira/browse/YARN-9674 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.2 >Reporter: ANANDA G B >Priority: Major > Attachments: RM_Issue.png > > > 'Max AM Resource' calculated for default partition using 'Effective Max > Capacity' and ohter partitions it using 'Effective Capacity'. > Which one is correct implemenation? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9674) AM Resource calculation is wrong
[ https://issues.apache.org/jira/browse/YARN-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-9674: - Summary: AM Resource calculation is wrong (was: 'Max AM Resource' calculated for default partition using 'Effective Max Capacity' and ohter partitions it using 'Effective Capacity') > AM Resource calculation is wrong > > > Key: YARN-9674 > URL: https://issues.apache.org/jira/browse/YARN-9674 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.2 >Reporter: ANANDA G B >Priority: Major > Attachments: RM_Issue.png > > > 'Max AM Resource' calculated for default partition using 'Effective Max > Capacity' and ohter partitions it using 'Effective Capacity'. > Which one is correct implemenation? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9674) Max AM Resource calculation is wrong
[ https://issues.apache.org/jira/browse/YARN-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-9674: - Summary: Max AM Resource calculation is wrong (was: AM Resource calculation is wrong) > Max AM Resource calculation is wrong > > > Key: YARN-9674 > URL: https://issues.apache.org/jira/browse/YARN-9674 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.2 >Reporter: ANANDA G B >Priority: Major > Attachments: RM_Issue.png > > > 'Max AM Resource' calculated for default partition using 'Effective Max > Capacity' and ohter partitions it using 'Effective Capacity'. > Which one is correct implemenation? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9674) 'Max AM Resource' calculated for default partition using 'Effective Max Capacity' and ohter partitions it using 'Effective Capacity'
ANANDA G B created YARN-9674: Summary: 'Max AM Resource' calculated for default partition using 'Effective Max Capacity' and ohter partitions it using 'Effective Capacity' Key: YARN-9674 URL: https://issues.apache.org/jira/browse/YARN-9674 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.1.2 Reporter: ANANDA G B Attachments: RM_Issue.png 'Max AM Resource' calculated for default partition using 'Effective Max Capacity' and ohter partitions it using 'Effective Capacity'. Which one is correct implemenation? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
