[jira] [Updated] (YARN-7330) Add support to show GPU on UI/metrics
[ https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7330: -- Attachment: YARN-7330.012.patch fixed js issues > Add support to show GPU on UI/metrics > - > > Key: YARN-7330 > URL: https://issues.apache.org/jira/browse/YARN-7330 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7330.0-wip.patch, YARN-7330.003.patch, > YARN-7330.004.patch, YARN-7330.006.patch, YARN-7330.007.patch, > YARN-7330.008.patch, YARN-7330.009.patch, YARN-7330.011.patch, > YARN-7330.012.patch, YARN-7330.1-wip.patch, YARN-7330.2-wip.patch, > screencapture-0-wip.png > > > We should be able to view GPU metrics from UI/REST API. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7533) Documentation for absolute resource support in CS
[ https://issues.apache.org/jira/browse/YARN-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7533: -- Attachment: YARN-7533-YARN-5881.002.patch Thanks [~eepayne] Attaching patch correcting this. Also in another note, YARN-7544 is focussing on removing the new added config items *min-resource* and *max-resource*. We will be using *capacity* and *max-capacity* itself. This could help users to configure as % or absolute values in same config parameter. cc/[~leftnoteasy] > Documentation for absolute resource support in CS > - > > Key: YARN-7533 > URL: https://issues.apache.org/jira/browse/YARN-7533 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7533-YARN-5881.002.patch, YARN-7533.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7544) Configuration cleanup associated to support absolute resources in a queue
[ https://issues.apache.org/jira/browse/YARN-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260762#comment-16260762 ] Sunil G commented on YARN-7544: --- YARN-7507 tracks test failures. Its unrelated to this patch. cc/ [~leftnoteasy] > Configuration cleanup associated to support absolute resources in a queue > - > > Key: YARN-7544 > URL: https://issues.apache.org/jira/browse/YARN-7544 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7544-YARN-5881.001.patch > > > Its better to reuse existing *capacity* and *max-capacity* instead of > introducing new config items. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7544) Configuration cleanup associated to support absolute resources in a queue
[ https://issues.apache.org/jira/browse/YARN-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7544: -- Attachment: YARN-7544-YARN-5881.001.patch v1 patch. > Configuration cleanup associated to support absolute resources in a queue > - > > Key: YARN-7544 > URL: https://issues.apache.org/jira/browse/YARN-7544 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7544-YARN-5881.001.patch > > > Its better to reuse existing *capacity* and *max-capacity* instead of > introducing new config items. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7544) Configuration cleanup associated to support absolute resources in a queue
[ https://issues.apache.org/jira/browse/YARN-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7544: -- Summary: Configuration cleanup associated to support absolute resources in a queue (was: Configuration cleanup associated to support absolute resources in a queuec) > Configuration cleanup associated to support absolute resources in a queue > - > > Key: YARN-7544 > URL: https://issues.apache.org/jira/browse/YARN-7544 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > > Its better to reuse existing *capacity* and *max-capacity* instead of > introducing new config items. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7544) Configuration cleanup associated to support absolute resources in a queuec
Sunil G created YARN-7544: - Summary: Configuration cleanup associated to support absolute resources in a queuec Key: YARN-7544 URL: https://issues.apache.org/jira/browse/YARN-7544 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G Its better to reuse existing *capacity* and *max-capacity* instead of introducing new config items. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7538) Performance improvement related to absolute min-max resources
[ https://issues.apache.org/jira/browse/YARN-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7538: -- Attachment: YARN-7538.001.patch v1 patch. > Performance improvement related to absolute min-max resources > - > > Key: YARN-7538 > URL: https://issues.apache.org/jira/browse/YARN-7538 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7538.001.patch > > > performance improvements. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7538) Performance improvement related to absolute min-max resources
Sunil G created YARN-7538: - Summary: Performance improvement related to absolute min-max resources Key: YARN-7538 URL: https://issues.apache.org/jira/browse/YARN-7538 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G performance improvements. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7533) Documentation for absolute resource support in CS
[ https://issues.apache.org/jira/browse/YARN-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7533: -- Attachment: YARN-7533.001.patch v1 patch > Documentation for absolute resource support in CS > - > > Key: YARN-7533 > URL: https://issues.apache.org/jira/browse/YARN-7533 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7533.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7480) Render tooltips on columns where text is clipped
[ https://issues.apache.org/jira/browse/YARN-7480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258876#comment-16258876 ] Sunil G commented on YARN-7480: --- ASF license changes needs fix. Kindly fix this. > Render tooltips on columns where text is clipped > > > Key: YARN-7480 > URL: https://issues.apache.org/jira/browse/YARN-7480 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: YARN-7480.001.patch, YARN-7480.002.patch > > > In em-table, when text gets clipped the information is lost. Need to render a > tooltip to show the full text in these cases -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7533) Documentation for absolute resource support in CS
Sunil G created YARN-7533: - Summary: Documentation for absolute resource support in CS Key: YARN-7533 URL: https://issues.apache.org/jira/browse/YARN-7533 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7510) Merge work for YARN-5881
[ https://issues.apache.org/jira/browse/YARN-7510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7510: -- Attachment: YARN-7510.002.patch branch is rebased and running jenkins again on branch patch (combined) > Merge work for YARN-5881 > > > Key: YARN-7510 > URL: https://issues.apache.org/jira/browse/YARN-7510 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7510.001.patch, YARN-7510.002.patch > > > Merge YARN-5881 work -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7469) Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit
[ https://issues.apache.org/jira/browse/YARN-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257011#comment-16257011 ] Sunil G commented on YARN-7469: --- Pushed to 2.9 as well. > Capacity Scheduler Intra-queue preemption: User can starve if newest app is > exactly at user limit > - > > Key: YARN-7469 > URL: https://issues.apache.org/jira/browse/YARN-7469 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, yarn >Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2 >Reporter: Eric Payne >Assignee: Eric Payne > Fix For: 2.8.3, 3.0.0, 3.1.0, 2.10.0, 2.9.1 > > Attachments: UnitTestToShowStarvedUser.patch, YARN-7469.001.patch > > > Queue Configuration: > - Total Memory: 20GB > - 2 Queues > -- Queue1 > --- Memory: 10GB > --- MULP: 10% > --- ULF: 2.0 > - Minimum Container Size: 0.5GB > Use Case: > - User1 submits app1 to Queue1 and consumes 20GB > - User2 submits app2 to Queue1 and requests 7.5GB > - Preemption monitor preempts 7.5GB from app1. Capacity Scheduler gives those > resources to User2 > - User 3 submits app3 to Queue1. To begin with, app3 is requesting 1 > container for the AM. > - Preemption monitor never preempts a container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7469) Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit
[ https://issues.apache.org/jira/browse/YARN-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7469: -- Fix Version/s: 2.9.1 > Capacity Scheduler Intra-queue preemption: User can starve if newest app is > exactly at user limit > - > > Key: YARN-7469 > URL: https://issues.apache.org/jira/browse/YARN-7469 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, yarn >Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2 >Reporter: Eric Payne >Assignee: Eric Payne > Fix For: 2.8.3, 3.0.0, 3.1.0, 2.10.0, 2.9.1 > > Attachments: UnitTestToShowStarvedUser.patch, YARN-7469.001.patch > > > Queue Configuration: > - Total Memory: 20GB > - 2 Queues > -- Queue1 > --- Memory: 10GB > --- MULP: 10% > --- ULF: 2.0 > - Minimum Container Size: 0.5GB > Use Case: > - User1 submits app1 to Queue1 and consumes 20GB > - User2 submits app2 to Queue1 and requests 7.5GB > - Preemption monitor preempts 7.5GB from app1. Capacity Scheduler gives those > resources to User2 > - User 3 submits app3 to Queue1. To begin with, app3 is requesting 1 > container for the AM. > - Preemption monitor never preempts a container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7469) Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit
[ https://issues.apache.org/jira/browse/YARN-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256279#comment-16256279 ] Sunil G commented on YARN-7469: --- Yes. :) I missed 2.9. Will backport now. > Capacity Scheduler Intra-queue preemption: User can starve if newest app is > exactly at user limit > - > > Key: YARN-7469 > URL: https://issues.apache.org/jira/browse/YARN-7469 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, yarn >Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2 >Reporter: Eric Payne >Assignee: Eric Payne > Fix For: 2.8.3, 3.0.0, 3.1.0, 2.10.0 > > Attachments: UnitTestToShowStarvedUser.patch, YARN-7469.001.patch > > > Queue Configuration: > - Total Memory: 20GB > - 2 Queues > -- Queue1 > --- Memory: 10GB > --- MULP: 10% > --- ULF: 2.0 > - Minimum Container Size: 0.5GB > Use Case: > - User1 submits app1 to Queue1 and consumes 20GB > - User2 submits app2 to Queue1 and requests 7.5GB > - Preemption monitor preempts 7.5GB from app1. Capacity Scheduler gives those > resources to User2 > - User 3 submits app3 to Queue1. To begin with, app3 is requesting 1 > container for the AM. > - Preemption monitor never preempts a container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7483) Test cases cleanup post YARN-5881
[ https://issues.apache.org/jira/browse/YARN-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7483: -- Attachment: YARN-7483-YARN-5881.001.patch renamed patch to apply to YARN-5881 branch > Test cases cleanup post YARN-5881 > - > > Key: YARN-7483 > URL: https://issues.apache.org/jira/browse/YARN-7483 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7483-YARN-5881.001.patch, YARN-7483.001.patch > > > Few UT cases has to be cleaned or rewritten cleanly to consider both > absolute/percentage based configuration. > Related test classes > # TestLeafQueue > # TestCapacityScheduler > # TestCapacitySchedulerWithMultiResourceTypes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7483) Test cases cleanup post YARN-5881
[ https://issues.apache.org/jira/browse/YARN-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7483: -- Attachment: YARN-7483.001.patch Attaching patch to clean up test case issues. cc/ [~leftnoteasy] please help to review. > Test cases cleanup post YARN-5881 > - > > Key: YARN-7483 > URL: https://issues.apache.org/jira/browse/YARN-7483 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7483.001.patch > > > Few UT cases has to be cleaned or rewritten cleanly to consider both > absolute/percentage based configuration. > Related test classes > # TestLeafQueue > # TestCapacityScheduler > # TestCapacitySchedulerWithMultiResourceTypes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7469) Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit
[ https://issues.apache.org/jira/browse/YARN-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255386#comment-16255386 ] Sunil G commented on YARN-7469: --- +1 Committing shortly. > Capacity Scheduler Intra-queue preemption: User can starve if newest app is > exactly at user limit > - > > Key: YARN-7469 > URL: https://issues.apache.org/jira/browse/YARN-7469 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, yarn >Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2 >Reporter: Eric Payne >Assignee: Eric Payne > Attachments: UnitTestToShowStarvedUser.patch, YARN-7469.001.patch > > > Queue Configuration: > - Total Memory: 20GB > - 2 Queues > -- Queue1 > --- Memory: 10GB > --- MULP: 10% > --- ULF: 2.0 > - Minimum Container Size: 0.5GB > Use Case: > - User1 submits app1 to Queue1 and consumes 20GB > - User2 submits app2 to Queue1 and requests 7.5GB > - Preemption monitor preempts 7.5GB from app1. Capacity Scheduler gives those > resources to User2 > - User 3 submits app3 to Queue1. To begin with, app3 is requesting 1 > container for the AM. > - Preemption monitor never preempts a container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7483) Test cases cleanup post YARN-5881
[ https://issues.apache.org/jira/browse/YARN-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7483: -- Description: Few UT cases has to be cleaned or rewritten cleanly to consider both absolute/percentage based configuration. Related test classes # TestLeafQueue # TestCapacityScheduler # TestCapacitySchedulerWithMultiResourceTypes was: Few UT cases has to be cleaned or rewritten cleanly to consider both absolute/percentage based configuration. Related test classes # TestLeafQueue # TestCapacityScheduler > Test cases cleanup post YARN-5881 > - > > Key: YARN-7483 > URL: https://issues.apache.org/jira/browse/YARN-7483 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > > Few UT cases has to be cleaned or rewritten cleanly to consider both > absolute/percentage based configuration. > Related test classes > # TestLeafQueue > # TestCapacityScheduler > # TestCapacitySchedulerWithMultiResourceTypes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7482) Max applications calculation per queue has to be retrospected with absolute resource support
[ https://issues.apache.org/jira/browse/YARN-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255229#comment-16255229 ] Sunil G commented on YARN-7482: --- Test case failure is not related to this patch. Its been tracked in YARN-7483 > Max applications calculation per queue has to be retrospected with absolute > resource support > > > Key: YARN-7482 > URL: https://issues.apache.org/jira/browse/YARN-7482 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7482-YARN-5881.001.patch > > > Retrospect max system applications w.r.t absolute resource configuration in a > queue -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7510) Merge work for YARN-5881
[ https://issues.apache.org/jira/browse/YARN-7510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7510: -- Attachment: YARN-7510.001.patch Attaching an initial branch patch for jenkins > Merge work for YARN-5881 > > > Key: YARN-7510 > URL: https://issues.apache.org/jira/browse/YARN-7510 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7510.001.patch > > > Merge YARN-5881 work -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7510) Merge work for YARN-5881
Sunil G created YARN-7510: - Summary: Merge work for YARN-5881 Key: YARN-7510 URL: https://issues.apache.org/jira/browse/YARN-7510 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G Merge YARN-5881 work -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7508) NPE in FiCaSchedulerApp when debug log enabled and try to commit outdated reserved proposal in async-scheduling mode
[
https://issues.apache.org/jira/browse/YARN-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255051#comment-16255051
]
Sunil G commented on YARN-7508:
---
I think its fine to change to
{{schedulerContainer.getSchedulerNode().getReservedContainer()}}
Is there any more instance of similar usage? If so we can move that also as per
this patch's proposal.
> NPE in FiCaSchedulerApp when debug log enabled and try to commit outdated
> reserved proposal in async-scheduling mode
>
>
> Key: YARN-7508
> URL: https://issues.apache.org/jira/browse/YARN-7508
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: Tao Yang
>Assignee: Tao Yang
> Attachments: YARN-7508.001.patch
>
>
> YARN-6678 have fixed the IllegalStateException problem but the debug log it
> added may cause NPE when trying to print containerId of non-existed reserved
> container on this node. Replace
> {{schedulerContainer.getSchedulerNode().getReservedContainer().getContainerId()}}
> with {{schedulerContainer.getSchedulerNode().getReservedContainer()}} can
> fix this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7482) Max applications calculation per queue has to be retrospected with absolute resource support
[ https://issues.apache.org/jira/browse/YARN-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7482: -- Attachment: YARN-7482-YARN-5881.001.patch v1 patch. This helps to fix the max application pblm in LeafQueue. cc/ [~rohithsharma] > Max applications calculation per queue has to be retrospected with absolute > resource support > > > Key: YARN-7482 > URL: https://issues.apache.org/jira/browse/YARN-7482 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7482-YARN-5881.001.patch > > > Retrospect max system applications w.r.t absolute resource configuration in a > queue -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7492) Set up SASS for new YARN UI styling
[ https://issues.apache.org/jira/browse/YARN-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7492: -- Summary: Set up SASS for new YARN UI styling (was: Set up SASS for UI styling) > Set up SASS for new YARN UI styling > --- > > Key: YARN-7492 > URL: https://issues.apache.org/jira/browse/YARN-7492 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: YARN-7492.001.patch, YARN-7492.002.patch, > YARN-7492.003.patch, YARN-7492.004.patch > > > SASS will help in improving the quality and maintainablity of our styles. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7492) Set up SASS for UI styling
[ https://issues.apache.org/jira/browse/YARN-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254875#comment-16254875 ] Sunil G commented on YARN-7492: --- +1 Committing shortly. > Set up SASS for UI styling > -- > > Key: YARN-7492 > URL: https://issues.apache.org/jira/browse/YARN-7492 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: YARN-7492.001.patch, YARN-7492.002.patch, > YARN-7492.003.patch, YARN-7492.004.patch > > > SASS will help in improving the quality and maintainablity of our styles. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7505) RM REST endpoints generate malformed JSON
[
https://issues.apache.org/jira/browse/YARN-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254698#comment-16254698
]
Sunil G commented on YARN-7505:
---
[~templedf], I agree that the first response which you mentioned with maps are
not created cleanly enough. However this will break few clients who upgrade to
latest version with this patch.
I think its better to skip for GA. Otherwise deprecated tags need to be
mentioned and keep old ones for while with new ones, and remove in next
minor/major version after marking with deprecated tag.
> RM REST endpoints generate malformed JSON
> -
>
> Key: YARN-7505
> URL: https://issues.apache.org/jira/browse/YARN-7505
> Project: Hadoop YARN
> Issue Type: Bug
> Components: restapi
>Affects Versions: 3.0.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-7505.001.patch, YARN-7505.002.patch
>
>
> For all endpoints that return DAOs that contain maps, the generated JSON is
> malformed. For example:
> % curl 'http://localhost:8088/ws/v1/cluster/apps'
> {"apps":{"app":[{"id":"application_1510777276702_0001","user":"daniel","name":"QuasiMonteCarlo","queue":"root.daniel","state":"RUNNING","finalStatus":"UNDEFINED","progress":5.0,"trackingUI":"ApplicationMaster","trackingUrl":"http://dhcp-10-16-0-181.pa.cloudera.com:8088/proxy/application_1510777276702_0001/","diagnostics":"","clusterId":1510777276702,"applicationType":"MAPREDUCE","applicationTags":"","priority":0,"startedTime":1510777317853,"finishedTime":0,"elapsedTime":21623,"amContainerLogs":"http://dhcp-10-16-0-181.pa.cloudera.com:8042/node/containerlogs/container_1510777276702_0001_01_01/daniel","amHostHttpAddress":"dhcp-10-16-0-181.pa.cloudera.com:8042","amRPCAddress":"dhcp-10-16-0-181.pa.cloudera.com:63371","allocatedMB":5120,"allocatedVCores":4,"reservedMB":0,"reservedVCores":0,"runningContainers":4,"memorySeconds":49820,"vcoreSeconds":26,"queueUsagePercentage":62.5,"clusterUsagePercentage":62.5,"resourceSecondsMap":{"entry":{"key":"test2","value":"0"},"entry":{"key":"test","value":"0"},"entry":{"key":"memory-mb","value":"49820"},"entry":{"key":"vcores","value":"26"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":0,"preemptedVcoreSeconds":0,"preemptedResourceSecondsMap":{},"resourceRequests":[{"priority":20,"resourceName":"dhcp-10-16-0-181.pa.cloudera.com","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"/default-rack","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"*","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false}],"logAggregationStatus":"DISABLED","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}}]}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7438) Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest / placement algorithm
[
https://issues.apache.org/jira/browse/YARN-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253818#comment-16253818
]
Sunil G commented on YARN-7438:
---
Thanks [~leftnoteasy] for the effort here.
Some general comments:
# lastPendingAsk and newPendingAsk seems a little tricky to understand and its
per Scheduler Key. Could we rename this better.
# A general doubt regarding {{ContainerRequest}}. Once we save set of resource
requests associated with a given container, it might have various resource
names (node local, rack local etc). But ANY will be common (accumulated count)
for all container demand on given Scheduler Key. This is a refactoring ticket,
so I just noticed this while a common class was created for same. Pls correct
me if I am wrong.
# lastPendingAsk could be cached?
> Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest
> / placement algorithm
> ---
>
> Key: YARN-7438
> URL: https://issues.apache.org/jira/browse/YARN-7438
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-7438.001.patch
>
>
> In additional to YARN-6040, we need to make changes to SchedulingPlacementSet
> to make it:
> 1) Agnostic to ResourceRequest (so once we have YARN-6592 merged, we can add
> new SchedulingPlacementSet implementation in parallel with
> LocalitySchedulingPlacementSet to use/manage new requests API)
> 2) Agnostic to placement algorithm (now it is bind to delayed scheduling, we
> should update APIs to make sure new placement algorithms such as complex
> placement algorithms can be implemented by using SchedulingPlacementSet).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7469) Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit
[ https://issues.apache.org/jira/browse/YARN-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253749#comment-16253749 ] Sunil G commented on YARN-7469: --- Thanks [~eepayne]. This makes sense. I am good with v1 patch. I could commit this later if its fine. > Capacity Scheduler Intra-queue preemption: User can starve if newest app is > exactly at user limit > - > > Key: YARN-7469 > URL: https://issues.apache.org/jira/browse/YARN-7469 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, yarn >Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2 >Reporter: Eric Payne >Assignee: Eric Payne > Attachments: UnitTestToShowStarvedUser.patch, YARN-7469.001.patch > > > Queue Configuration: > - Total Memory: 20GB > - 2 Queues > -- Queue1 > --- Memory: 10GB > --- MULP: 10% > --- ULF: 2.0 > - Minimum Container Size: 0.5GB > Use Case: > - User1 submits app1 to Queue1 and consumes 20GB > - User2 submits app2 to Queue1 and requests 7.5GB > - Preemption monitor preempts 7.5GB from app1. Capacity Scheduler gives those > resources to User2 > - User 3 submits app3 to Queue1. To begin with, app3 is requesting 1 > container for the AM. > - Preemption monitor never preempts a container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7092) [YARN-3368] Log viewer in application page in yarn-ui-v2
[ https://issues.apache.org/jira/browse/YARN-7092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253497#comment-16253497 ] Sunil G commented on YARN-7092: --- [~akhilpb], could you please help to rebase so tht we can push this to each app's page. > [YARN-3368] Log viewer in application page in yarn-ui-v2 > > > Key: YARN-7092 > URL: https://issues.apache.org/jira/browse/YARN-7092 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Akhil PB >Assignee: Akhil PB > Attachments: YARN-7092.001.patch > > > Feature to view application logs in new yarn-ui. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6651) Flow Activity should specify 'metricstoretrieve' in its query to ATSv2 to retrieve CPU and memory
[
https://issues.apache.org/jira/browse/YARN-6651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253496#comment-16253496
]
Sunil G commented on YARN-6651:
---
[~akhilpb], I discussed with [~rohithsharma] also.
We can use "?metricstoretrieve=YARN_APPLICATION_CPU,YARN_APPLICATION_MEMORY"
instead of "fields=ALL" so that CPU and MEMORY could retrieved and updated in
FLOW RUNS page.
YARN-6653 is similar but was specific to apps CPU and MEMORY usage, We can use
same filter to query metrics for an app in a FLOW RUN
> Flow Activity should specify 'metricstoretrieve' in its query to ATSv2 to
> retrieve CPU and memory
> --
>
> Key: YARN-6651
> URL: https://issues.apache.org/jira/browse/YARN-6651
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: Haibo Chen
>Assignee: Akhil PB
>
> When you click on Flow Acitivity => \{a flow\} => flow runs, the web server
> sends a REST query to ATSv2 TimelineReaderServer, but it does not include a
> query param 'metricstoretrieve" to get any metrics back.
> Instead, we should add
> '?metricstoretrieve=YARN_APPLICATION_CPU,YARN_APPLICATION_MEMORY' to the
> query to get CPU and MEMORY back.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7330) Add support to show GPU on UI/metrics
[
https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-7330:
--
Attachment: YARN-7330.009.patch
Attaching v9 patch after changing {{resourceToSimplifiedUnit}} in converter.js.
In this patch. we convert value to a common base unit. [~leftnoteasy], is this
fine? could you please help to check the js changes.
> Add support to show GPU on UI/metrics
> -
>
> Key: YARN-7330
> URL: https://issues.apache.org/jira/browse/YARN-7330
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: YARN-7330.0-wip.patch, YARN-7330.003.patch,
> YARN-7330.004.patch, YARN-7330.006.patch, YARN-7330.007.patch,
> YARN-7330.008.patch, YARN-7330.009.patch, YARN-7330.1-wip.patch,
> YARN-7330.2-wip.patch, screencapture-0-wip.png
>
>
> We should be able to view GPU metrics from UI/REST API.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7464) Introduce filters in Nodes page of new YARN UI
[ https://issues.apache.org/jira/browse/YARN-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7464: -- Summary: Introduce filters in Nodes page of new YARN UI (was: Allow fiters on Nodes page) > Introduce filters in Nodes page of new YARN UI > --- > > Key: YARN-7464 > URL: https://issues.apache.org/jira/browse/YARN-7464 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-08 at 4.56.04 PM.png, Screen Shot > 2017-11-08 at 4.56.12 PM.png, YARN-7464.001.patch, YARN-7464.002.patch, > YARN-7464.003.patch, YARN-7464.004.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7492) Set up SASS for UI styling
[ https://issues.apache.org/jira/browse/YARN-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253076#comment-16253076 ] Sunil G commented on YARN-7492: --- Please revert configs.env local changes > Set up SASS for UI styling > -- > > Key: YARN-7492 > URL: https://issues.apache.org/jira/browse/YARN-7492 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: YARN-7492.001.patch, YARN-7492.002.patch, > YARN-7492.003.patch > > > SASS will help in improving the quality and maintainablity of our styles. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7462) Render outstanding resource requests on application page of new YARN UI
[ https://issues.apache.org/jira/browse/YARN-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7462: -- Summary: Render outstanding resource requests on application page of new YARN UI (was: Render outstanding resource requests on application details page) > Render outstanding resource requests on application page of new YARN UI > --- > > Key: YARN-7462 > URL: https://issues.apache.org/jira/browse/YARN-7462 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-08 at 3.24.30 PM.png, Screen Shot > 2017-11-08 at 3.38.48 PM.png, YARN-7462.001.patch, YARN-7462.002.patch, > YARN-7462.003.patch, YARN-7462.004.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7330) Add support to show GPU on UI/metrics
[ https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251770#comment-16251770 ] Sunil G commented on YARN-7330: --- [~skmvasu] ,could you please take a look > Add support to show GPU on UI/metrics > - > > Key: YARN-7330 > URL: https://issues.apache.org/jira/browse/YARN-7330 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7330.0-wip.patch, YARN-7330.003.patch, > YARN-7330.004.patch, YARN-7330.006.patch, YARN-7330.007.patch, > YARN-7330.008.patch, YARN-7330.1-wip.patch, YARN-7330.2-wip.patch, > screencapture-0-wip.png > > > We should be able to view GPU metrics from UI/REST API. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7330) Add support to show GPU on UI/metrics
[ https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7330: -- Attachment: YARN-7330.008.patch Fixing compilation issue. > Add support to show GPU on UI/metrics > - > > Key: YARN-7330 > URL: https://issues.apache.org/jira/browse/YARN-7330 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7330.0-wip.patch, YARN-7330.003.patch, > YARN-7330.004.patch, YARN-7330.006.patch, YARN-7330.007.patch, > YARN-7330.008.patch, YARN-7330.1-wip.patch, YARN-7330.2-wip.patch, > screencapture-0-wip.png > > > We should be able to view GPU metrics from UI/REST API. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7330) Add support to show GPU on UI/metrics
[ https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7330: -- Attachment: YARN-7330.007.patch Attaching patch addressing comment from Vasudevan. > Add support to show GPU on UI/metrics > - > > Key: YARN-7330 > URL: https://issues.apache.org/jira/browse/YARN-7330 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7330.0-wip.patch, YARN-7330.003.patch, > YARN-7330.004.patch, YARN-7330.006.patch, YARN-7330.007.patch, > YARN-7330.1-wip.patch, YARN-7330.2-wip.patch, screencapture-0-wip.png > > > We should be able to view GPU metrics from UI/REST API. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7464) Allow fiters on Nodes page
[ https://issues.apache.org/jira/browse/YARN-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251528#comment-16251528 ] Sunil G commented on YARN-7464: --- [~skmvasu], seems like the patch is not getting applied to trunk. I think yarn-node-managers was renamed to yarn-node-status and causing this. Could you please help to rebase this.. > Allow fiters on Nodes page > -- > > Key: YARN-7464 > URL: https://issues.apache.org/jira/browse/YARN-7464 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-08 at 4.56.04 PM.png, Screen Shot > 2017-11-08 at 4.56.12 PM.png, YARN-7464.001.patch, YARN-7464.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7494) Add muti node lookup support for better placement
Sunil G created YARN-7494: - Summary: Add muti node lookup support for better placement Key: YARN-7494 URL: https://issues.apache.org/jira/browse/YARN-7494 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G Instead of single node, for effectiveness we can consider a multi node lookup based on partition to start with. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7462) Render outstanding resource requests on application details page
[ https://issues.apache.org/jira/browse/YARN-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251228#comment-16251228 ] Sunil G commented on YARN-7462: --- pending jenkins > Render outstanding resource requests on application details page > > > Key: YARN-7462 > URL: https://issues.apache.org/jira/browse/YARN-7462 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-08 at 3.24.30 PM.png, Screen Shot > 2017-11-08 at 3.38.48 PM.png, YARN-7462.001.patch, YARN-7462.002.patch, > YARN-7462.003.patch, YARN-7462.004.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7464) Allow fiters on Nodes page
[ https://issues.apache.org/jira/browse/YARN-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251230#comment-16251230 ] Sunil G commented on YARN-7464: --- +1 pending jenkins for commit. > Allow fiters on Nodes page > -- > > Key: YARN-7464 > URL: https://issues.apache.org/jira/browse/YARN-7464 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-08 at 4.56.04 PM.png, Screen Shot > 2017-11-08 at 4.56.12 PM.png, YARN-7464.001.patch, YARN-7464.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7462) Render outstanding resource requests on application details page
[ https://issues.apache.org/jira/browse/YARN-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251058#comment-16251058 ] Sunil G commented on YARN-7462: --- v3 patch looks good to me. Committing later today if there are no objections. > Render outstanding resource requests on application details page > > > Key: YARN-7462 > URL: https://issues.apache.org/jira/browse/YARN-7462 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-08 at 3.24.30 PM.png, Screen Shot > 2017-11-08 at 3.38.48 PM.png, YARN-7462.001.patch, YARN-7462.002.patch, > YARN-7462.003.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7464) Allow fiters on Nodes page
[ https://issues.apache.org/jira/browse/YARN-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250995#comment-16250995 ] Sunil G commented on YARN-7464: --- Please rename "Node Managers" to "Nodes Status" > Allow fiters on Nodes page > -- > > Key: YARN-7464 > URL: https://issues.apache.org/jira/browse/YARN-7464 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-08 at 4.56.04 PM.png, Screen Shot > 2017-11-08 at 4.56.12 PM.png, YARN-7464.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7462) Render outstanding resource requests on application details page
[
https://issues.apache.org/jira/browse/YARN-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250868#comment-16250868
]
Sunil G commented on YARN-7462:
---
[~skmvasu] Thanks for the patch.
I have one issue in this patch. Post YARN-3926, we support more resource types
other than memory and vcores.
But REST endpoint
org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ResourceInfo is not
having multiple resource support in a clean way. So post this patch, we need to
raise a common ticket in UI to show resources other than memory and cpu also.
Some minor comments:
# {{.history/}} is added to git ignore list. Why this is needed?
# Rename {{Priority}} to {{Scheduler Key}}
# {{ResourceName}} to {{Resource Name}}. A space to be added for better
readability
# {{NumContainers}}. Could we say like {{# Containers}} as it indicates a count.
# {{Outstanding resource requests}} to {{Outstanding Resource Requests}}
# If {{model.app.resourceRequests}} is empty, could we hide the whole table?
> Render outstanding resource requests on application details page
>
>
> Key: YARN-7462
> URL: https://issues.apache.org/jira/browse/YARN-7462
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Vasudevan Skm
>Assignee: Vasudevan Skm
> Attachments: Screen Shot 2017-11-08 at 3.24.30 PM.png, Screen Shot
> 2017-11-08 at 3.38.48 PM.png, YARN-7462.001.patch, YARN-7462.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7411) Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee
[
https://issues.apache.org/jira/browse/YARN-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250668#comment-16250668
]
Sunil G commented on YARN-7411:
---
Thanks [~leftnoteasy] for review and commit. Test case changed as suggested
will be handled in YARN-7483
> Inter-Queue preemption's computeFixpointAllocation need to handle absolute
> resources while computing normalizedGuarantee
>
>
> Key: YARN-7411
> URL: https://issues.apache.org/jira/browse/YARN-7411
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-5881
>Reporter: Sunil G
>Assignee: Sunil G
> Fix For: YARN-5881
>
> Attachments: YARN-7411-YARN-5881.004.patch,
> YARN-7411-YARN-5881.005.patch, YARN-7411.001.patch,
> YARN-7441.YARN-5881.002.patch, YARN-7441.YARN-5881.003.patch
>
>
> {{normalizedGuarantee}} is computed based on queue's capacity. This has to be
> updated correctly when CS starts to accept queue's capacity in terms of
> absolute resource.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7469) Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit
[ https://issues.apache.org/jira/browse/YARN-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249884#comment-16249884 ] Sunil G commented on YARN-7469: --- Hi [~eepayne] This is a nice catch and nasty one to debug too. I think the proposed patch helps to solve the issue. In broader perspective, i think we are lacking dead zone here. In a way, now min container is the dead zone here. But if user gets more control on this, may be more oscillations could be avoided. May be we can take up that also in another ticket. > Capacity Scheduler Intra-queue preemption: User can starve if newest app is > exactly at user limit > - > > Key: YARN-7469 > URL: https://issues.apache.org/jira/browse/YARN-7469 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, yarn >Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2 >Reporter: Eric Payne >Assignee: Eric Payne > Attachments: UnitTestToShowStarvedUser.patch, YARN-7469.001.patch > > > Queue Configuration: > - Total Memory: 20GB > - 2 Queues > -- Queue1 > --- Memory: 10GB > --- MULP: 10% > --- ULF: 2.0 > - Minimum Container Size: 0.5GB > Use Case: > - User1 submits app1 to Queue1 and consumes 20GB > - User2 submits app2 to Queue1 and requests 7.5GB > - Preemption monitor preempts 7.5GB from app1. Capacity Scheduler gives those > resources to User2 > - User 3 submits app3 to Queue1. To begin with, app3 is requesting 1 > container for the AM. > - Preemption monitor never preempts a container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6918) Remove acls after queue delete to avoid memory leak
[ https://issues.apache.org/jira/browse/YARN-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249857#comment-16249857 ] Sunil G commented on YARN-6918: --- Thanks [~bibinchundatt] for raising this. Is this a major leak which happens in normal code path? Is it critical to 2.9.0 also ? cc/ [~subru] [~asuresh] [~leftnoteasy] > Remove acls after queue delete to avoid memory leak > --- > > Key: YARN-6918 > URL: https://issues.apache.org/jira/browse/YARN-6918 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Critical > Attachments: YARN-6918.001.patch > > > Acl for deleted queue need to removed from allAcls to avoid leak > (Priority,YarnAuthorizer) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249725#comment-16249725 ] Sunil G commented on YARN-7159: --- [~maniraj...@gmail.com] This patch needs rebase, Kindly help to check the same. > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch, > YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch, > YARN-7159.009.patch, YARN-7159.010.patch, YARN-7159.011.patch, > YARN-7159.012.patch, YARN-7159.013.patch, YARN-7159.015.patch, > YARN-7159.016.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249711#comment-16249711 ] Sunil G commented on YARN-7159: --- Committing this patch shortly. > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch, > YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch, > YARN-7159.009.patch, YARN-7159.010.patch, YARN-7159.011.patch, > YARN-7159.012.patch, YARN-7159.013.patch, YARN-7159.015.patch, > YARN-7159.016.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7483) Test cases cleanup post YARN-5881
Sunil G created YARN-7483: - Summary: Test cases cleanup post YARN-5881 Key: YARN-7483 URL: https://issues.apache.org/jira/browse/YARN-7483 Project: Hadoop YARN Issue Type: Sub-task Components: test Affects Versions: YARN-5881 Reporter: Sunil G Assignee: Sunil G Few UT cases has to be cleaned or rewritten cleanly to consider both absolute/percentage based configuration. Related test classes # TestLeafQueue # TestCapacityScheduler -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7482) Max applications calculation per queue has to be retrospected with absolute resource support
Sunil G created YARN-7482: - Summary: Max applications calculation per queue has to be retrospected with absolute resource support Key: YARN-7482 URL: https://issues.apache.org/jira/browse/YARN-7482 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Affects Versions: YARN-5881 Reporter: Sunil G Assignee: Sunil G Retrospect max system applications w.r.t absolute resource configuration in a queue -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6651) Flow Activity should specify 'metricstoretrieve' in its query to ATSv2 to retrieve CPU and memory
[
https://issues.apache.org/jira/browse/YARN-6651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249654#comment-16249654
]
Sunil G commented on YARN-6651:
---
I think we are sending ALL option to every field from ATS. [~akhilpb], could
you please confirm. cc/[~rohithsharma]
> Flow Activity should specify 'metricstoretrieve' in its query to ATSv2 to
> retrieve CPU and memory
> --
>
> Key: YARN-6651
> URL: https://issues.apache.org/jira/browse/YARN-6651
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: Haibo Chen
>Assignee: Akhil PB
>
> When you click on Flow Acitivity => \{a flow\} => flow runs, the web server
> sends a REST query to ATSv2 TimelineReaderServer, but it does not include a
> query param 'metricstoretrieve" to get any metrics back.
> Instead, we should add
> '?metricstoretrieve=YARN_APPLICATION_CPU,YARN_APPLICATION_MEMORY' to the
> query to get CPU and MEMORY back.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7445) Render Applications and Services page with filters in new YARN UI
[ https://issues.apache.org/jira/browse/YARN-7445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7445: -- Summary: Render Applications and Services page with filters in new YARN UI (was: Render filters on Applications and Services page) > Render Applications and Services page with filters in new YARN UI > - > > Key: YARN-7445 > URL: https://issues.apache.org/jira/browse/YARN-7445 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-06 at 5.59.06 PM.png, > YARN-7445.001.patch, YARN-7445.002.patch, YARN-7445.003.patch, > YARN-7445.004.patch, YARN-7445.005.patch > > > Filters are missing on the new webui -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7445) Render filters on Applications and Services page
[ https://issues.apache.org/jira/browse/YARN-7445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249635#comment-16249635 ] Sunil G commented on YARN-7445: --- I am committing this with pixel issue in Progress column as open. As confirmed by [~skmvasu], this need some skin changes and will addressed in a common jira. Kindly confirm > Render filters on Applications and Services page > > > Key: YARN-7445 > URL: https://issues.apache.org/jira/browse/YARN-7445 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-06 at 5.59.06 PM.png, > YARN-7445.001.patch, YARN-7445.002.patch, YARN-7445.003.patch, > YARN-7445.004.patch, YARN-7445.005.patch > > > Filters are missing on the new webui -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7119) yarn rmadmin -updateNodeResource should be updated for resource types
[ https://issues.apache.org/jira/browse/YARN-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249629#comment-16249629 ] Sunil G commented on YARN-7119: --- [~templedf] could you please help to take a look at final patch > yarn rmadmin -updateNodeResource should be updated for resource types > - > > Key: YARN-7119 > URL: https://issues.apache.org/jira/browse/YARN-7119 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Manikandan R > Attachments: YARN-7119.001.patch, YARN-7119.002.patch, > YARN-7119.002.patch, YARN-7119.003.patch, YARN-7119.004.patch, > YARN-7119.004.patch, YARN-7119.005.patch, YARN-7119.006.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6627) Use deployed webapp folder to launch new YARN UI
[ https://issues.apache.org/jira/browse/YARN-6627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-6627. --- Resolution: Fixed Fix Version/s: 2.9.0 Thanks [~asuresh] for backporting this to 2.9.0 (ref: d15be3c93a0d03d15eb2407d266680343b51bf15). Closing this ticket by updating fix version. > Use deployed webapp folder to launch new YARN UI > > > Key: YARN-6627 > URL: https://issues.apache.org/jira/browse/YARN-6627 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Sunil G >Assignee: Sunil G > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: YARN-6627.0001.patch, YARN-6627.0002.patch, > YARN-6627.0003.patch, YARN-6627.branch-2.001.patch > > > Currently new ui war file is placed in share/hadoop/yarn folder. Along with > this, its better to have ui2 folder placed under share/hadoop/yarn/webapps so > that UI could be launched from a defined folder. > This will also help to make some cutom config related to ui as well. > cc/[~jianhe] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7475) Fix Container log link in new YARN UI
[ https://issues.apache.org/jira/browse/YARN-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249069#comment-16249069 ] Sunil G commented on YARN-7475: --- Thanks [~subru] for review and commit. Thanks [~elgoiri] for additional reviews. > Fix Container log link in new YARN UI > - > > Key: YARN-7475 > URL: https://issues.apache.org/jira/browse/YARN-7475 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Sunil G >Assignee: Sunil G > Fix For: 2.9.0, 3.0.0 > > Attachments: YARN-7475.001.patch > > > Container log link is broken -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7475) container log link is not working in new YARN UI
[ https://issues.apache.org/jira/browse/YARN-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248912#comment-16248912 ] Sunil G commented on YARN-7475: --- Thanks [~goirix]. In adapter side of yarn-ui, we have a common abstract.js class which updates host/namespace etc based on server(RM/NM/ATS) and its respective REST end point namespace. Hence a common way for host and namespace was added. Yes, some of adapters are not overridden from abstract class such as yarn-container-log etc. We planned for a general cleanup to create a better abstract class for all, and its a clean up task pending. I think current solution clearly fixes bug, and I think its better to handle all this clean up which u mentioned to be handled as a common cleanup jira as its not a functionality change or behavior cc/ [~leftnoteasy] > container log link is not working in new YARN UI > > > Key: YARN-7475 > URL: https://issues.apache.org/jira/browse/YARN-7475 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7475.001.patch > > > Container log link is broken -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6627) Use deployed webapp folder to launch new YARN UI
[ https://issues.apache.org/jira/browse/YARN-6627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-6627: -- Attachment: YARN-6627.branch-2.001.patch Attaching branch-2 patch. cc/ [~vrushalic] [~subru] [~rohithsharma] > Use deployed webapp folder to launch new YARN UI > > > Key: YARN-6627 > URL: https://issues.apache.org/jira/browse/YARN-6627 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Sunil G >Assignee: Sunil G > Fix For: 3.0.0-alpha4 > > Attachments: YARN-6627.0001.patch, YARN-6627.0002.patch, > YARN-6627.0003.patch, YARN-6627.branch-2.001.patch > > > Currently new ui war file is placed in share/hadoop/yarn folder. Along with > this, its better to have ui2 folder placed under share/hadoop/yarn/webapps so > that UI could be launched from a defined folder. > This will also help to make some cutom config related to ui as well. > cc/[~jianhe] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-6627) Use deployed webapp folder to launch new YARN UI
[ https://issues.apache.org/jira/browse/YARN-6627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G reopened YARN-6627: --- reopening to attach branch-2 patch > Use deployed webapp folder to launch new YARN UI > > > Key: YARN-6627 > URL: https://issues.apache.org/jira/browse/YARN-6627 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Sunil G >Assignee: Sunil G > Fix For: 3.0.0-alpha4 > > Attachments: YARN-6627.0001.patch, YARN-6627.0002.patch, > YARN-6627.0003.patch > > > Currently new ui war file is placed in share/hadoop/yarn folder. Along with > this, its better to have ui2 folder placed under share/hadoop/yarn/webapps so > that UI could be launched from a defined folder. > This will also help to make some cutom config related to ui as well. > cc/[~jianhe] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7475) container log link is not working in new YARN UI
[ https://issues.apache.org/jira/browse/YARN-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7475: -- Attachment: YARN-7475.001.patch v1 patch > container log link is not working in new YARN UI > > > Key: YARN-7475 > URL: https://issues.apache.org/jira/browse/YARN-7475 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7475.001.patch > > > Container log link is broken -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Moved] (YARN-7475) container log link is not working in new YARN UI
[ https://issues.apache.org/jira/browse/YARN-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G moved MAPREDUCE-7005 to YARN-7475: -- Affects Version/s: (was: 3.0.0-beta1) (was: 2.9.0) 2.9.0 3.0.0-beta1 Target Version/s: 2.9.0, 3.0.0 (was: 2.9.0, 3.0.0) Key: YARN-7475 (was: MAPREDUCE-7005) Project: Hadoop YARN (was: Hadoop Map/Reduce) > container log link is not working in new YARN UI > > > Key: YARN-7475 > URL: https://issues.apache.org/jira/browse/YARN-7475 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-beta1, 2.9.0 >Reporter: Sunil G >Assignee: Sunil G > > Container log link is broken -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7452) Decommissioning node default value to be zero in new YARN UI
[ https://issues.apache.org/jira/browse/YARN-7452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7452: -- Summary: Decommissioning node default value to be zero in new YARN UI (was: Decommissioning node: NaN issue) > Decommissioning node default value to be zero in new YARN UI > > > Key: YARN-7452 > URL: https://issues.apache.org/jira/browse/YARN-7452 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm >Priority: Trivial > Attachments: YARN-7452.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7445) Render filters on Applications and Services page
[ https://issues.apache.org/jira/browse/YARN-7445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248466#comment-16248466 ] Sunil G commented on YARN-7445: --- Thanks [~skmvasu] for addressing comments Kicking jenkins manually. Will commit later today pending jenkins and if there are no objections > Render filters on Applications and Services page > > > Key: YARN-7445 > URL: https://issues.apache.org/jira/browse/YARN-7445 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-06 at 5.59.06 PM.png, > YARN-7445.001.patch, YARN-7445.002.patch, YARN-7445.003.patch, > YARN-7445.004.patch > > > Filters are missing on the new webui -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7411) Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee
[
https://issues.apache.org/jira/browse/YARN-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-7411:
--
Attachment: YARN-7411-YARN-5881.005.patch
YARN-5881 branch is rebased to trunk.
Also rebasing latest patch to run jenkins.
> Inter-Queue preemption's computeFixpointAllocation need to handle absolute
> resources while computing normalizedGuarantee
>
>
> Key: YARN-7411
> URL: https://issues.apache.org/jira/browse/YARN-7411
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-5881
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-7411-YARN-5881.004.patch,
> YARN-7411-YARN-5881.005.patch, YARN-7411.001.patch,
> YARN-7441.YARN-5881.002.patch, YARN-7441.YARN-5881.003.patch
>
>
> {{normalizedGuarantee}} is computed based on queue's capacity. This has to be
> updated correctly when CS starts to accept queue's capacity in terms of
> absolute resource.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7330) Add support to show GPU on UI/metrics
[
https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247864#comment-16247864
]
Sunil G commented on YARN-7330:
---
Thanks [~leftnoteasy]
# It might better to use {{ResourceUtils.getResourcesTypeInfo()}} for DAO
classes instead of getAllResourcesListCopy
# In {{NMGpuResourceInfo}}, assignedGpuDevices will only give device minor
number and we can know that this GPU is used. Could we also give which
app/container is using this ?
I am good with JS changes for UI.
> Add support to show GPU on UI/metrics
> -
>
> Key: YARN-7330
> URL: https://issues.apache.org/jira/browse/YARN-7330
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: YARN-7330.0-wip.patch, YARN-7330.003.patch,
> YARN-7330.004.patch, YARN-7330.1-wip.patch, YARN-7330.2-wip.patch,
> screencapture-0-wip.png
>
>
> We should be able to view GPU metrics from UI/REST API.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5881) Enable configuration of queue capacity in terms of absolute resources
[ https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-5881: -- Attachment: (was: YARN-7411-YARN-5881.004.patch) > Enable configuration of queue capacity in terms of absolute resources > - > > Key: YARN-5881 > URL: https://issues.apache.org/jira/browse/YARN-5881 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sean Po >Assignee: Wangda Tan > Attachments: > YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf, > YARN-5881.v0.patch, YARN-5881.v1.patch > > > Currently, Yarn RM supports the configuration of queue capacity in terms of a > proportion to cluster capacity. In the context of Yarn being used as a public > cloud service, it makes more sense if queues can be configured absolutely. > This will allow administrators to set usage limits more concretely and > simplify customer expectations for cluster allocation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5881) Enable configuration of queue capacity in terms of absolute resources
[ https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-5881: -- Attachment: YARN-7411-YARN-5881.004.patch Adding testcase. > Enable configuration of queue capacity in terms of absolute resources > - > > Key: YARN-5881 > URL: https://issues.apache.org/jira/browse/YARN-5881 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sean Po >Assignee: Wangda Tan > Attachments: > YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf, > YARN-5881.v0.patch, YARN-5881.v1.patch, YARN-7411-YARN-5881.004.patch > > > Currently, Yarn RM supports the configuration of queue capacity in terms of a > proportion to cluster capacity. In the context of Yarn being used as a public > cloud service, it makes more sense if queues can be configured absolutely. > This will allow administrators to set usage limits more concretely and > simplify customer expectations for cluster allocation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7411) Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee
[
https://issues.apache.org/jira/browse/YARN-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-7411:
--
Attachment: YARN-7411-YARN-5881.004.patch
Adding a test case to check multi resources in preemption. Thanks
[~leftnoteasy], kindly help to check the same.
> Inter-Queue preemption's computeFixpointAllocation need to handle absolute
> resources while computing normalizedGuarantee
>
>
> Key: YARN-7411
> URL: https://issues.apache.org/jira/browse/YARN-7411
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-5881
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-7411-YARN-5881.004.patch, YARN-7411.001.patch,
> YARN-7441.YARN-5881.002.patch, YARN-7441.YARN-5881.003.patch
>
>
> {{normalizedGuarantee}} is computed based on queue's capacity. This has to be
> updated correctly when CS starts to accept queue's capacity in terms of
> absolute resource.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7369) Improve the resource types docs
[ https://issues.apache.org/jira/browse/YARN-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246874#comment-16246874 ] Sunil G edited comment on YARN-7369 at 11/10/17 1:34 AM: - patch5 is missing ResourceModel.md file changes was (Author: sunilg): patch5 is missing ResourceModeal.md file changes > Improve the resource types docs > --- > > Key: YARN-7369 > URL: https://issues.apache.org/jira/browse/YARN-7369 > Project: Hadoop YARN > Issue Type: Sub-task > Components: docs >Affects Versions: 3.1.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: YARN-7369.001.patch, YARN-7369.002.patch, > YARN-7369.003.patch, YARN-7369.004.patch, YARN-7369.005.patch, > YARN-7369.branch-3.0.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7369) Improve the resource types docs
[ https://issues.apache.org/jira/browse/YARN-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246874#comment-16246874 ] Sunil G commented on YARN-7369: --- patch5 is missing ResourceModeal.md file changes > Improve the resource types docs > --- > > Key: YARN-7369 > URL: https://issues.apache.org/jira/browse/YARN-7369 > Project: Hadoop YARN > Issue Type: Sub-task > Components: docs >Affects Versions: 3.1.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: YARN-7369.001.patch, YARN-7369.002.patch, > YARN-7369.003.patch, YARN-7369.004.patch, YARN-7369.005.patch, > YARN-7369.branch-3.0.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7330) Add support to show GPU on UI/metrics
[ https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246123#comment-16246123 ] Sunil G commented on YARN-7330: --- Thank you, I am checking this and will share my comments if any. > Add support to show GPU on UI/metrics > - > > Key: YARN-7330 > URL: https://issues.apache.org/jira/browse/YARN-7330 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7330.0-wip.patch, YARN-7330.003.patch, > YARN-7330.004.patch, YARN-7330.1-wip.patch, YARN-7330.2-wip.patch, > screencapture-0-wip.png > > > We should be able to view GPU metrics from UI/REST API. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7119) yarn rmadmin -updateNodeResource should be updated for resource types
[ https://issues.apache.org/jira/browse/YARN-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246084#comment-16246084 ] Sunil G commented on YARN-7119: --- Looks good in general. I will wait for couple of days for other folks also to take a look. cc/ [~leftnoteasy] [~templedf] > yarn rmadmin -updateNodeResource should be updated for resource types > - > > Key: YARN-7119 > URL: https://issues.apache.org/jira/browse/YARN-7119 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Manikandan R > Attachments: YARN-7119.001.patch, YARN-7119.002.patch, > YARN-7119.002.patch, YARN-7119.003.patch, YARN-7119.004.patch, > YARN-7119.004.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246083#comment-16246083 ] Sunil G commented on YARN-7159: --- Thanks. I am good with latest patch. [~leftnoteasy] [~templedf], any comments on latest patch ? > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch, > YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch, > YARN-7159.009.patch, YARN-7159.010.patch, YARN-7159.011.patch, > YARN-7159.012.patch, YARN-7159.013.patch, YARN-7159.015.patch, > YARN-7159.016.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6909) The performance advantages of YARN-6679 are lost when resource types are used
[
https://issues.apache.org/jira/browse/YARN-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246068#comment-16246068
]
Sunil G edited comment on YARN-6909 at 11/9/17 5:08 PM:
Thanks [~leftnoteasy]. Attaching new patch after fixing test cases
was (Author: sunilg):
Fixing test cases
> The performance advantages of YARN-6679 are lost when resource types are used
> -
>
> Key: YARN-6909
> URL: https://issues.apache.org/jira/browse/YARN-6909
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Sunil G
>Priority: Critical
> Attachments: YARN-6909.001.patch, YARN-6909.002.patch,
> YARN-6909.003.patch, YARN-6909.004.patch, YARN-6909.005.patch,
> YARN-6909.006.patch
>
>
> YARN-6679 added the {{SimpleResource}} as a lightweight replacement for
> {{ResourcePBImpl}} when a protobuf isn't needed. With resource types enabled
> and anything other than memory and CPU defined, {{ResourcePBImpl}} will
> always be used.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6909) The performance advantages of YARN-6679 are lost when resource types are used
[
https://issues.apache.org/jira/browse/YARN-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-6909:
--
Attachment: YARN-6909.006.patch
Fixing test cases
> The performance advantages of YARN-6679 are lost when resource types are used
> -
>
> Key: YARN-6909
> URL: https://issues.apache.org/jira/browse/YARN-6909
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Sunil G
>Priority: Critical
> Attachments: YARN-6909.001.patch, YARN-6909.002.patch,
> YARN-6909.003.patch, YARN-6909.004.patch, YARN-6909.005.patch,
> YARN-6909.006.patch
>
>
> YARN-6679 added the {{SimpleResource}} as a lightweight replacement for
> {{ResourcePBImpl}} when a protobuf isn't needed. With resource types enabled
> and anything other than memory and CPU defined, {{ResourcePBImpl}} will
> always be used.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7452) Decommissioning node: NaN issue
[ https://issues.apache.org/jira/browse/YARN-7452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245431#comment-16245431 ] Sunil G commented on YARN-7452: --- Patch looks straight forward. Committing shortly if no objections. > Decommissioning node: NaN issue > --- > > Key: YARN-7452 > URL: https://issues.apache.org/jira/browse/YARN-7452 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: YARN-7452.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7452) Decommissioning node: NaN issue
[ https://issues.apache.org/jira/browse/YARN-7452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7452: -- Priority: Trivial (was: Major) > Decommissioning node: NaN issue > --- > > Key: YARN-7452 > URL: https://issues.apache.org/jira/browse/YARN-7452 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm >Priority: Trivial > Attachments: YARN-7452.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7445) Render filters on Applications and Services page
[
https://issues.apache.org/jira/browse/YARN-7445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245427#comment-16245427
]
Sunil G commented on YARN-7445:
---
Patch looks fine to me.
[~vasduevanskm], Progress column has a rendering issue earlier where it looses
one pixel in each row. If not done, please include the same in this patch.
Few minor nits.
# pls add *alasql.js* in LICENSE.txt
# {{headerTitle: 'Status'}} should be changed to {{headerTitle: 'State'}}
# For *state* columns, midWidth could be 70 or 75.
> Render filters on Applications and Services page
>
>
> Key: YARN-7445
> URL: https://issues.apache.org/jira/browse/YARN-7445
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Vasudevan Skm
>Assignee: Vasudevan Skm
> Attachments: Screen Shot 2017-11-06 at 5.59.06 PM.png,
> YARN-7445.001.patch, YARN-7445.002.patch, YARN-7445.003.patch
>
>
> Filters are missing on the new webui
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7330) Add support to show GPU on UI/metrics
[ https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244289#comment-16244289 ] Sunil G commented on YARN-7330: --- cc/ [~skmvasu] > Add support to show GPU on UI/metrics > - > > Key: YARN-7330 > URL: https://issues.apache.org/jira/browse/YARN-7330 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7330.0-wip.patch, YARN-7330.003.patch, > YARN-7330.004.patch, YARN-7330.1-wip.patch, YARN-7330.2-wip.patch, > screencapture-0-wip.png > > > We should be able to view GPU metrics from UI/REST API. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7453) RM fail to switch to active after first successful start
[
https://issues.apache.org/jira/browse/YARN-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-7453:
--
Attachment: YARN-7453.001.patch
Reverted YARN-6840's ResourceManager and ZKRMStateStore changes. This solves
the issue for now. Detailed analysis will be shared a bit later
> RM fail to switch to active after first successful start
>
>
> Key: YARN-7453
> URL: https://issues.apache.org/jira/browse/YARN-7453
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0, 3.1.0
>Reporter: Rohith Sharma K S
>Priority: Blocker
> Attachments: YARN-7453.001.patch
>
>
> It is observed that RM fail to switch to ACTIVE after first successful start!
> The below exception throws when RM is switching from ACTIVE->STANDBY->ACTIVE.
> This continues in loop!
> {noformat}
> 2017-11-07 15:08:11,664 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning
> to active state
> 2017-11-07 15:08:11,669 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Recovery
> started
> 2017-11-07 15:08:11,669 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Loaded
> RM state version info 1.5
> 2017-11-07 15:08:11,670 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Failed to
> load/recover state
> org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1006)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl.doOperation(CuratorTransactionImpl.java:159)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl.access$200(CuratorTransactionImpl.java:44)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl$2.call(CuratorTransactionImpl.java:129)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl$2.call(CuratorTransactionImpl.java:125)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl.commit(CuratorTransactionImpl.java:122)
> at
> org.apache.hadoop.util.curator.ZKCuratorManager$SafeTransaction.commit(ZKCuratorManager.java:403)
> at
> org.apache.hadoop.util.curator.ZKCuratorManager.safeSetData(ZKCuratorManager.java:372)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.getAndIncrementEpoch(ZKRMStateStore.java:493)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:771)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1162)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1202)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1198)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1198)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:320)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:894)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:473)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:607)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6909) The performance advantages of YARN-6679 are lost when resource types are used
[
https://issues.apache.org/jira/browse/YARN-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-6909:
--
Attachment: YARN-6909.005.patch
Thanks [~templedf]. Fixed all comments, except below.
bq.Shouldn't both the newDefaultInformation() be methods belong to the
ResourceInformation class?
We have seen a bit for performance dip when using same api from
ResourceInformation. I will again check and see how far is off now.
> The performance advantages of YARN-6679 are lost when resource types are used
> -
>
> Key: YARN-6909
> URL: https://issues.apache.org/jira/browse/YARN-6909
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Sunil G
>Priority: Critical
> Attachments: YARN-6909.001.patch, YARN-6909.002.patch,
> YARN-6909.003.patch, YARN-6909.004.patch, YARN-6909.005.patch
>
>
> YARN-6679 added the {{SimpleResource}} as a lightweight replacement for
> {{ResourcePBImpl}} when a protobuf isn't needed. With resource types enabled
> and anything other than memory and CPU defined, {{ResourcePBImpl}} will
> always be used.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7427) NullPointerException in ResourceInfo when queue has not used label
[
https://issues.apache.org/jira/browse/YARN-7427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237323#comment-16237323
]
Sunil G commented on YARN-7427:
---
I think this address corner case where a queue is not used a portion of
labelled resource. I think the getter of DAO class could now skip the if
condition of checking for resources is null or not.
> NullPointerException in ResourceInfo when queue has not used label
> --
>
> Key: YARN-7427
> URL: https://issues.apache.org/jira/browse/YARN-7427
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-7427.001.patch
>
>
> {noformat}Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ResourceInfo.toString(ResourceInfo.java:65)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(CapacitySchedulerPage.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderLeafQueueInfoWithPartition(CapacitySchedulerPage.java:107)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.render(CapacitySchedulerPage.java:96)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:43)
> at
> org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl$EImp._v(HamletImpl.java:117)
> at
> org.apache.hadoop.yarn.webapp.hamlet2.Hamlet$LI.__(Hamlet.java:7709)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueueBlock.render(CapacitySchedulerPage.java:301)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:43)
> at
> org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl$EImp._v(HamletImpl.java:117)
> at
> org.apache.hadoop.yarn.webapp.hamlet2.Hamlet$LI.__(Hamlet.java:7709)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueuesBlock.render(CapacitySchedulerPage.java:470)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
> at
> org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl$EImp._v(HamletImpl.java:117)
> at org.apache.hadoop.yarn.webapp.hamlet2.Hamlet$TD.__(Hamlet.java:848)
> at
> org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:71)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
> at
> org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:212)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RmController.scheduler(RmController.java:86)
> ... 56 more{noformat}
> For example, configure: {noformat}
> yarn.scheduler.capacity.root.queues
> default,a
>
>
> yarn.scheduler.capacity.root.accessible-node-labels
> x
>
>
> yarn.scheduler.capacity.root.default.accessible-node-labels
> x
>
>
>
> yarn.scheduler.capacity.root.default.accessible-node-labels.x.maximum-capacity
> 100
>
>
>
> yarn.scheduler.capacity.root.default.accessible-node-labels.x.capacity
> 100
> {noformat}
> , then the above exception is thrown when refreshing the scheduler UI
> (/cluster/scheduler)
> As a result the queue block dropdowns cannot be minimized.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[
https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237300#comment-16237300
]
Sunil G commented on YARN-7159:
---
[~maniraj...@gmail.com] Also one more thing. I tested performance UT test also
with this patch. Not much regression. However please run these tests against
and compare with this patch also. So we can see how much performance
improvement this patch is bringing.
{{mvn test
-Dtest=TestCapacitySchedulerPerf#testUserLimitThroughputForTwoResources
-DRunCapacitySchedulerPerfTests=true}}
> Normalize unit of resource objects in RM and avoid to do unit conversion in
> critical path
> -
>
> Key: YARN-7159
> URL: https://issues.apache.org/jira/browse/YARN-7159
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Manikandan R
>Priority: Critical
> Attachments: YARN-7159.001.patch, YARN-7159.002.patch,
> YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch,
> YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch,
> YARN-7159.009.patch, YARN-7159.010.patch, YARN-7159.011.patch
>
>
> Currently resource conversion could happen in critical code path when
> different unit is specified by client. This could impact performance and
> throughput of RM a lot. We should do unit normalization when resource passed
> to RM and avoid expensive unit conversion every time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[
https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237297#comment-16237297
]
Sunil G commented on YARN-7159:
---
Thanks [~maniraj...@gmail.com]
Few minor comments
# In {{ResourceUtils#addResourceInformation}}, else condition could be avoided.
Below code snippet could be common outside once *units* is updated inside if
conditions.
{code}
510 nodeResources.get(resourceType).setValue(resourceValue);
511 nodeResources.get(resourceType).setUnits(units);
{code}
# In {{MockNM}}, {{getResource}} could be more generic enough to get whats
should be the units and value for different resources.
# getRegNMResponse -> getRegisteredNMResponse
> Normalize unit of resource objects in RM and avoid to do unit conversion in
> critical path
> -
>
> Key: YARN-7159
> URL: https://issues.apache.org/jira/browse/YARN-7159
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Manikandan R
>Priority: Critical
> Attachments: YARN-7159.001.patch, YARN-7159.002.patch,
> YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch,
> YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch,
> YARN-7159.009.patch, YARN-7159.010.patch, YARN-7159.011.patch
>
>
> Currently resource conversion could happen in critical code path when
> different unit is specified by client. This could impact performance and
> throughput of RM a lot. We should do unit normalization when resource passed
> to RM and avoid expensive unit conversion every time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6909) The performance advantages of YARN-6679 are lost when resource types are used
[
https://issues.apache.org/jira/browse/YARN-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-6909:
--
Attachment: YARN-6909.004.patch
Thanks [~leftnoteasy]. Attaching latest patch to address the comments.
> The performance advantages of YARN-6679 are lost when resource types are used
> -
>
> Key: YARN-6909
> URL: https://issues.apache.org/jira/browse/YARN-6909
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Sunil G
>Priority: Critical
> Attachments: YARN-6909.001.patch, YARN-6909.002.patch,
> YARN-6909.003.patch, YARN-6909.004.patch
>
>
> YARN-6679 added the {{SimpleResource}} as a lightweight replacement for
> {{ResourcePBImpl}} when a protobuf isn't needed. With resource types enabled
> and anything other than memory and CPU defined, {{ResourcePBImpl}} will
> always be used.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7392) Render cluster information on new YARN web ui
[ https://issues.apache.org/jira/browse/YARN-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7392: -- Summary: Render cluster information on new YARN web ui (was: Render cluster information on the new yarn web ui) > Render cluster information on new YARN web ui > - > > Key: YARN-7392 > URL: https://issues.apache.org/jira/browse/YARN-7392 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm >Priority: Major > Attachments: YARN-7392.001.cluster.info.patch, YARN-7392.001.patch, > YARN-7392.002.patch, YARN-7392.003.patch, YARN-7392.004.patch, > YARN-7392.005.patch, YARN-7392.006.patch, patch 003.png, screenshot-1.png, > with bootstrap toolti.png > > > Cluster information is missing in the new web UI. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7392) Render cluster information on the new yarn web ui
[ https://issues.apache.org/jira/browse/YARN-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236967#comment-16236967 ] Sunil G commented on YARN-7392: --- +1 to latest patch. > Render cluster information on the new yarn web ui > - > > Key: YARN-7392 > URL: https://issues.apache.org/jira/browse/YARN-7392 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm >Priority: Major > Attachments: YARN-7392.001.cluster.info.patch, YARN-7392.001.patch, > YARN-7392.002.patch, YARN-7392.003.patch, YARN-7392.004.patch, > YARN-7392.005.patch, YARN-7392.006.patch, patch 003.png, screenshot-1.png, > with bootstrap toolti.png > > > Cluster information is missing in the new web UI. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7411) Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee
[
https://issues.apache.org/jira/browse/YARN-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-7411:
--
Attachment: YARN-7441.YARN-5881.003.patch
Patch needed rebase, Attaching v3
> Inter-Queue preemption's computeFixpointAllocation need to handle absolute
> resources while computing normalizedGuarantee
>
>
> Key: YARN-7411
> URL: https://issues.apache.org/jira/browse/YARN-7411
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-5881
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Major
> Attachments: YARN-7411.001.patch, YARN-7441.YARN-5881.002.patch,
> YARN-7441.YARN-5881.003.patch
>
>
> {{normalizedGuarantee}} is computed based on queue's capacity. This has to be
> updated correctly when CS starts to accept queue's capacity in terms of
> absolute resource.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7364) Queue dash board in new YARN UI has incorrect values
[ https://issues.apache.org/jira/browse/YARN-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236959#comment-16236959 ] Sunil G commented on YARN-7364: --- Thanks [~leftnoteasy] for helping to review and commit the same. > Queue dash board in new YARN UI has incorrect values > > > Key: YARN-7364 > URL: https://issues.apache.org/jira/browse/YARN-7364 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 3.0.0-beta1 >Reporter: Sunil G >Assignee: Sunil G >Priority: Critical > Fix For: 2.9.0, 3.0.0 > > Attachments: YARN-7364.001.patch, YARN-7364.002.patch > > > Queue dashboard in cluster over page does not show queue metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7410) Cleanup FixedValueResource to avoid dependency to ResourceUtils
[ https://issues.apache.org/jira/browse/YARN-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236940#comment-16236940 ] Sunil G commented on YARN-7410: --- Committing shortly. Thanks [~leftnoteasy] and [~templedf] > Cleanup FixedValueResource to avoid dependency to ResourceUtils > --- > > Key: YARN-7410 > URL: https://issues.apache.org/jira/browse/YARN-7410 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.0 >Reporter: Sunil G >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-7410.001.patch, YARN-7410.002.patch, > YARN-7410.003.patch > > > After YARN-7307, Client/AM don't need to keep a up-to-dated resource-type.xml > in the classpath. Instead, they can use YarnClient/ApplicationMasterProtocol > APIs to get the resource types from RM and refresh local types. > One biggest issue of this approach is FixedValueResource: Since we initialize > FixedValueResource in static block, and they won't be updated if resource > types refreshed. > So we need to properly update FixedValueResource to make it can get > up-to-date results -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7411) Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee
[
https://issues.apache.org/jira/browse/YARN-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-7411:
--
Attachment: YARN-7441.YARN-5881.002.patch
Attaching v2 patch after correcting test errors and comments from [~leftnoteasy]
Test cases are covered as per resource types cases. I will check how
effectively I can add per absolute resource context.
> Inter-Queue preemption's computeFixpointAllocation need to handle absolute
> resources while computing normalizedGuarantee
>
>
> Key: YARN-7411
> URL: https://issues.apache.org/jira/browse/YARN-7411
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-5881
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Major
> Attachments: YARN-7411.001.patch, YARN-7441.YARN-5881.002.patch
>
>
> {{normalizedGuarantee}} is computed based on queue's capacity. This has to be
> updated correctly when CS starts to accept queue's capacity in terms of
> absolute resource.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236040#comment-16236040
]
Sunil G commented on YARN-7370:
---
[~eepayne] Latest patch is fine to me. Pls help to commit.
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7370.001.patch, YARN-7370.002.patch,
> YARN-7370.003.patch, YARN-7370.004.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235898#comment-16235898
]
Sunil G commented on YARN-7370:
---
Thanks [~GergelyNovak]
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7370.001.patch, YARN-7370.002.patch,
> YARN-7370.003.patch, YARN-7370.004.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7410) Cleanup FixedValueResource to avoid dependency to ResourceUtils
[ https://issues.apache.org/jira/browse/YARN-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235867#comment-16235867 ] Sunil G commented on YARN-7410: --- +1 to latest patch. > Cleanup FixedValueResource to avoid dependency to ResourceUtils > --- > > Key: YARN-7410 > URL: https://issues.apache.org/jira/browse/YARN-7410 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.0 >Reporter: Sunil G >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-7410.001.patch, YARN-7410.002.patch, > YARN-7410.003.patch > > > After YARN-7307, Client/AM don't need to keep a up-to-dated resource-type.xml > in the classpath. Instead, they can use YarnClient/ApplicationMasterProtocol > APIs to get the resource types from RM and refresh local types. > One biggest issue of this approach is FixedValueResource: Since we initialize > FixedValueResource in static block, and they won't be updated if resource > types refreshed. > So we need to properly update FixedValueResource to make it can get > up-to-date results -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7424) Capacity Scheduler Intra-queue preemption: add property to only preempt up to configured MULP
[
https://issues.apache.org/jira/browse/YARN-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235857#comment-16235857
]
Sunil G commented on YARN-7424:
---
Thanks [~eepayne] for detailed analysis.
{{"yarn.resourcemanager.monitor.capacity.preemption.intra-queue-preemption.minimum-threshold"}}
could be configured to start intra queue preemption on a queue. But yes, this
is generic in all queues.
{{max-allowable-limit}} helps only to control preemption in a given round of
preemption calculation. This could be configured to a very low value so that
only few resource will be preempted in such cases.
Now the solution which you mentioned will help to control preemption. In a
cluster where we have free space and some users who are over MULP, i think we
can restrict such users to raise demand for preemption. In which preemption
mode, this will be added ? will it be common for both ?
> Capacity Scheduler Intra-queue preemption: add property to only preempt up to
> configured MULP
> -
>
> Key: YARN-7424
> URL: https://issues.apache.org/jira/browse/YARN-7424
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 3.0.0-beta1, 2.8.2
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
>
> If the queue's configured minimum user limit percent (MULP) is something
> small like 1%, all users will max out well over their MULP until 100 users
> have apps in the queue. Since the intra-queue preemption monitor tries to
> balance the resource among the users, most of the time in this use case it
> will be preempting containers on behalf of users that are already over their
> MULP guarantee.
> This JIRA proposes that a property should be provided so that a queue can be
> configured to only preempt on behalf of a user until that user has reached
> its MULP.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235826#comment-16235826
]
Sunil G commented on YARN-7370:
---
Sorry for jumping in little late here. Thanks for the effort [~GergelyNovak]
and [~eepayne]. Patch seems fine.
One quick doubt. In {{ProportionalCapacityPreemptionPolicy}}, LOG is printed
when {{csConfig}} is not null. So how could we quickly verify which config got
changed?
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7370.001.patch, YARN-7370.002.patch,
> YARN-7370.003.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6909) The performance advantages of YARN-6679 are lost when resource types are used
[
https://issues.apache.org/jira/browse/YARN-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-6909:
--
Attachment: YARN-6909.003.patch
Thanks [~leftnoteasy]. Attaching new patch after addressing all comments.
With this patch, I am getting below performance improvements
#ResourceTypes = 2. Avg of fastest 20: 57401.66
#ResourceTypes = 5. Avg of fastest 20: 50175.312
Earlier I was getting ~40k for 5 resource types.
> The performance advantages of YARN-6679 are lost when resource types are used
> -
>
> Key: YARN-6909
> URL: https://issues.apache.org/jira/browse/YARN-6909
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Sunil G
>Priority: Critical
> Attachments: YARN-6909.001.patch, YARN-6909.002.patch,
> YARN-6909.003.patch
>
>
> YARN-6679 added the {{SimpleResource}} as a lightweight replacement for
> {{ResourcePBImpl}} when a protobuf isn't needed. With resource types enabled
> and anything other than memory and CPU defined, {{ResourcePBImpl}} will
> always be used.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7364) Queue dash board in new YARN UI has incorrect values
[ https://issues.apache.org/jira/browse/YARN-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7364: -- Target Version/s: 2.9.0, 3.0.0 (was: 3.0.0) > Queue dash board in new YARN UI has incorrect values > > > Key: YARN-7364 > URL: https://issues.apache.org/jira/browse/YARN-7364 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 3.0.0-beta1 >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > Attachments: YARN-7364.001.patch, YARN-7364.002.patch > > > Queue dashboard in cluster over page does not show queue metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
