[jira] [Created] (YARN-10247) Application priority queue ACLs are not respected
Sunil G created YARN-10247: -- Summary: Application priority queue ACLs are not respected Key: YARN-10247 URL: https://issues.apache.org/jira/browse/YARN-10247 Project: Hadoop YARN Issue Type: Task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G This is a regression from queue path jira. App priority acls are not working correctly. {code:java} yarn.scheduler.capacity.root.B.acl_application_max_priority=[user=john group=users max_priority=4] {code} max_priority enforcement is not working. For user john, maximum supported priority is 4. However I can submit like priority 6 for this user. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10540) Node page is broken in YARN UI1 and UI2 including RMWebService api for nodes
Sunil G created YARN-10540: -- Summary: Node page is broken in YARN UI1 and UI2 including RMWebService api for nodes Key: YARN-10540 URL: https://issues.apache.org/jira/browse/YARN-10540 Project: Hadoop YARN Issue Type: Task Components: webapp Affects Versions: 3.2.2 Reporter: Sunil G YARN-10450 added changes in NodeInfo class. Various exceptions are showing while accessing UI2 and UI1 NODE pages. {code:java} Caused by: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.NodeInfo.(NodeInfo.java:103) at org.apache.hadoop.yarn.server.resourcemanager.webapp.NodesPage$NodesBlock.render(NodesPage.java:164) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79) at org.apache.hadoop.yarn.webapp.View.render(View.java:243) at org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49) at org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl$EImp._v(HamletImpl.java:117) at org.apache.hadoop.yarn.webapp.hamlet2.Hamlet$TD.__(Hamlet.java:848) at org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:71) at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82) at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:216) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RmController.nodes(RmController.java:70) {code} {code:java} 2020-12-19 22:55:54,846 WARN org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.NodeInfo.(NodeInfo.java:103) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.getNodes(RMWebServices.java:450) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5698) [YARN-3368] Launch new YARN UI under hadoop web app port
Sunil G created YARN-5698: - Summary: [YARN-3368] Launch new YARN UI under hadoop web app port Key: YARN-5698 URL: https://issues.apache.org/jira/browse/YARN-5698 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G Assignee: Sunil G As discussed in YARN-5145, it will be better to launch new web ui as a new webapp under same old port. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5705) Add support for Timeline V2 to new web UI
Sunil G created YARN-5705: - Summary: Add support for Timeline V2 to new web UI Key: YARN-5705 URL: https://issues.apache.org/jira/browse/YARN-5705 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G Assignee: Akhil PB Integrate timeline v2 to YARN-3368. This is a clone JIRA for YARN-4097 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5731) Preemption does not work in few corner cases where reservations are placed
Sunil G created YARN-5731: - Summary: Preemption does not work in few corner cases where reservations are placed Key: YARN-5731 URL: https://issues.apache.org/jira/browse/YARN-5731 Project: Hadoop YARN Issue Type: Bug Components: capacity scheduler Affects Versions: 2.7.3 Reporter: Sunil G Assignee: Sunil G Preemption doesnt kick in under below scenario. Two queues A and B each with 50% capacity and 100% maximum capacity and user limit factor 2. Job which is submitted to queueA has taken 95% of resources in cluster. Remaining 5% was also reserved by same job as demand was still higher. Now submit a small job with AM container size is lesser to above mentioned 5%. Job waits and no preemption is happening. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5785) [YARN-3368] Accessing applications and containers list from Node page is throwing few exceptions in console
Sunil G created YARN-5785: - Summary: [YARN-3368] Accessing applications and containers list from Node page is throwing few exceptions in console Key: YARN-5785 URL: https://issues.apache.org/jira/browse/YARN-5785 Project: Hadoop YARN Issue Type: Sub-task Components: yarn-ui-v2 Reporter: Sunil G Assignee: Akhil PB In node page, "List of Applications" and "List of Containers" links are causing few error logs in console. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5804) New UI2 is not able to launch with jetty 9 upgrade post HADOOP-10075
Sunil G created YARN-5804: - Summary: New UI2 is not able to launch with jetty 9 upgrade post HADOOP-10075 Key: YARN-5804 URL: https://issues.apache.org/jira/browse/YARN-5804 Project: Hadoop YARN Issue Type: Sub-task Components: yarn-ui-v2 Reporter: Sunil G Assignee: Sunil G Post HADOOP-10075, few compilation errors popped up. This jira is to track these problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5494) Nodes page throws "Sorry Error Occurred" message
[ https://issues.apache.org/jira/browse/YARN-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-5494. --- Resolution: Duplicate Closing this as duplicate of YARN-5785 > Nodes page throws "Sorry Error Occurred" message > > > Key: YARN-5494 > URL: https://issues.apache.org/jira/browse/YARN-5494 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Akhil PB >Priority: Critical > > Steps to reproduce: > * Click on Nodes. This page will list nodes of the cluster > * Click on one the nodes such as node1. ( It will redirect to > http://:4200/#/yarn-node/:31924/:8042 url) > This url prompts "Sorry Error Occurred" error. > {code}jquery.js:8630 XMLHttpRequest cannot load > http://xxx:xxx:8042/ws/v1/node. Cross origin requests are only supported for > protocol schemes: http, data, chrome, chrome-extension, https, > chrome-extension-resource.send @ jquery.js:8630 > ember.debug.js:30877 Error: Adapter operation failed > at new Error (native) > at Error.EmberError (http://xxx:4200/assets/vendor.js:25278:21) > at Error.ember$data$lib$adapters$errors$$AdapterError > (http://xxx:4200/assets/vendor.js:91198:50) > at Class.handleResponse (http://xxx:4200/assets/vendor.js:92494:16) > at Class.hash.error (http://xxx:4200/assets/vendor.js:92574:33) > at fire (http://xxx:4200/assets/vendor.js:3306:30) > at Object.fireWith [as rejectWith] > (http://xxx:4200/assets/vendor.js:3418:7) > at done (http://xxx:4200/assets/vendor.js:8473:14) > at XMLHttpRequest. (http://xxx:4200/assets/vendor.js:8806:9) > at Object.send (http://xxx:4200/assets/vendor.js:8837:10)onerrorDefault @ > ember.debug.js:30877{code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5618) Support for Intra queue preemption framework
[ https://issues.apache.org/jira/browse/YARN-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-5618. --- Resolution: Fixed Closing this Jira as DONE as this change also went along with YARN-2009 > Support for Intra queue preemption framework > > > Key: YARN-5618 > URL: https://issues.apache.org/jira/browse/YARN-5618 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Sunil G >Assignee: Sunil G > > Currently inter-queue preemption framework covers the basics (configs and > scheduling monitor interval etc). This new framework will come as new > CandidateSelector policy. Priority and user-limit will be a part of this > framework. > This is a tracking jira for the framework impl alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5618) Support for Intra queue preemption framework
[ https://issues.apache.org/jira/browse/YARN-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-5618. --- Resolution: Done > Support for Intra queue preemption framework > > > Key: YARN-5618 > URL: https://issues.apache.org/jira/browse/YARN-5618 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Sunil G >Assignee: Sunil G > > Currently inter-queue preemption framework covers the basics (configs and > scheduling monitor interval etc). This new framework will come as new > CandidateSelector policy. Priority and user-limit will be a part of this > framework. > This is a tracking jira for the framework impl alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5825) ProportionalPreemptionalPolicy could use readLock over LeafQueue instead of synchronized block
Sunil G created YARN-5825: - Summary: ProportionalPreemptionalPolicy could use readLock over LeafQueue instead of synchronized block Key: YARN-5825 URL: https://issues.apache.org/jira/browse/YARN-5825 Project: Hadoop YARN Issue Type: Bug Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G Currently in PCPP, {{synchronized (curQueue)}} is used in various places. Such instances could be replaced with a read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5865) Retrospect updateApplicationPriority api to handle state store exception in align with YARN-5611
Sunil G created YARN-5865: - Summary: Retrospect updateApplicationPriority api to handle state store exception in align with YARN-5611 Key: YARN-5865 URL: https://issues.apache.org/jira/browse/YARN-5865 Project: Hadoop YARN Issue Type: Bug Reporter: Sunil G Assignee: Sunil G Post YARN-5611, revisit dynamic update of application priority logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5868) pre-commit build shows random error from npm plugin related to yarn ui
Sunil G created YARN-5868: - Summary: pre-commit build shows random error from npm plugin related to yarn ui Key: YARN-5868 URL: https://issues.apache.org/jira/browse/YARN-5868 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Reporter: Sunil G Assignee: Sunil G As per below report https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/patch-compile-root.txt there s a random issue generated by 'npm install' during build. This jira is to track the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5869) Introduce Shuffle Speed metrics to identify slow nodes in cluster
Sunil G created YARN-5869: - Summary: Introduce Shuffle Speed metrics to identify slow nodes in cluster Key: YARN-5869 URL: https://issues.apache.org/jira/browse/YARN-5869 Project: Hadoop YARN Issue Type: Improvement Reporter: Sunil G Assignee: Sunil G This jira proposes new shuffle speed metrics which could help to identify slow nodes in the cluster. Existing Shuffle Handler in NM could help to get this metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5889) Improve user-limit calculation in capacity scheduler
Sunil G created YARN-5889: - Summary: Improve user-limit calculation in capacity scheduler Key: YARN-5889 URL: https://issues.apache.org/jira/browse/YARN-5889 Project: Hadoop YARN Issue Type: Bug Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G Currently user-limit is computed during every heartbeat allocation cycle with a write lock. To improve performance, this tickets is focussing on moving user-limit calculation out of heartbeat allocation flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5932) Retrospect moveToQueue in align with YARN-5611
Sunil G created YARN-5932: - Summary: Retrospect moveToQueue in align with YARN-5611 Key: YARN-5932 URL: https://issues.apache.org/jira/browse/YARN-5932 Project: Hadoop YARN Issue Type: Bug Components: capacity scheduler, resourcemanager Reporter: Sunil G Assignee: Sunil G All dynamic api's of an application's state change could follow a general design approach. Currently priority and app timeouts are following this approach all corner cases. *Steps* - Do a pre-validate check to ensure that changes are fine. - Update this information to state-store - Perform real move operation and update in-memory data structures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5984) Refactor move application across queue's CS level implementation
Sunil G created YARN-5984: - Summary: Refactor move application across queue's CS level implementation Key: YARN-5984 URL: https://issues.apache.org/jira/browse/YARN-5984 Project: Hadoop YARN Issue Type: Bug Components: capacity scheduler, resourcemanager Reporter: Sunil G Assignee: Sunil G Currently we use a top level write lock in CS#moveApplication. Also we are using few submission time apis in move. This jira will be focussing on coming up with a cleaner implementation for moveApplication and could try to share code with FS as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6001) Improve moveApplicationQueues command line
Sunil G created YARN-6001: - Summary: Improve moveApplicationQueues command line Key: YARN-6001 URL: https://issues.apache.org/jira/browse/YARN-6001 Project: Hadoop YARN Issue Type: Bug Components: client Reporter: Sunil G Assignee: Sunil G As an example, below command is used to move an application across queue. {noformat}./yarn application -movetoqueue application_1479894790219_0002 -queue default{noformat} Inline with other application's attribute modification features such as updateTimeout or priority, movetoqueue cli command lacks unification and more complex or error prone. Suggesting below cli command which can be used. {noformat} ./yarn application -appId application_1479894790219_0002 -move default {noformat} This is inline with other commands such as {noformat} ./yarn application -appId application_1479894790219_0002 -updatePriority 8 ./yarn application -appId application_1479894790219_0002 -updateLifetime 10 {noformat} Old command could be still there, but deprecated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5275) Timeline application page cannot be loaded when no application submitted/running on the cluster after HADOOP-9613
[ https://issues.apache.org/jira/browse/YARN-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-5275. --- Resolution: Not A Problem > Timeline application page cannot be loaded when no application > submitted/running on the cluster after HADOOP-9613 > - > > Key: YARN-5275 > URL: https://issues.apache.org/jira/browse/YARN-5275 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha1 >Reporter: Tsuyoshi Ozawa >Priority: Critical > > After HADOOP-9613, Timeline Web UI has a problem reported by [~leftnoteasy] > and [~sunilg] > {quote} > when no application submitted/running on the cluster, applications page > cannot be loaded. > {quote} > We should investigate the reason and fix it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6082) Webservice connection gets cutoff when it has to send back a large response (webservice)
Sunil G created YARN-6082: - Summary: Webservice connection gets cutoff when it has to send back a large response (webservice) Key: YARN-6082 URL: https://issues.apache.org/jira/browse/YARN-6082 Project: Hadoop YARN Issue Type: Bug Reporter: Sunil G Priority: Critical {noformat} 2017-01-11 07:17:11,475 WARN ipc.Server (Server.java:run(2202)) - Large response size 4476919 for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 172.27.0.101:39950 Call#951474 Retry#0 {noformat} In such cases, json output will get cutoff and client will not get clean response. For eg: {noformat} Unexpected token I in JSON at position 851 {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6129) Relook at FifoIntraQueuePreemptionPlugin#calculateIdealAssignedResourcePerApp parameter list
Sunil G created YARN-6129: - Summary: Relook at FifoIntraQueuePreemptionPlugin#calculateIdealAssignedResourcePerApp parameter list Key: YARN-6129 URL: https://issues.apache.org/jira/browse/YARN-6129 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler preemption Reporter: Sunil G Assignee: Sunil G Priority: Minor Revisit all argument list of FifoIntraQueuePreemptionPlugin#calculateIdealAssignedResourcePerApp as per [comment|https://issues.apache.org/jira/browse/YARN-5889?focusedCommentId=15843343&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15843343] from [~leftnoteasy] in YARN-5889. {{partitionBasedResource}} seems not used, but revisit whether its really needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6201) Add node label as a query param for nodes REST end point
Sunil G created YARN-6201: - Summary: Add node label as a query param for nodes REST end point Key: YARN-6201 URL: https://issues.apache.org/jira/browse/YARN-6201 Project: Hadoop YARN Issue Type: Bug Components: webapp Reporter: Sunil G Assignee: Sunil G For {{/nodes}} end point, label could be a query params. This could help to retrieve Node informations for a give label (with resources etc). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6215) TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in trunk
Sunil G created YARN-6215: - Summary: TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues fails in trunk Key: YARN-6215 URL: https://issues.apache.org/jira/browse/YARN-6215 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Sunil G *Error Message* Incorrect number of containers on the greedy app expected:<4> but was:<8> Failed test case [link|https://builds.apache.org/job/PreCommit-YARN-Build/15038/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair/TestFairSchedulerPreemption/testPreemptionBetweenNonSiblingQueues_FairSharePreemptionWithDRF_/] *Stacktrace* java.lang.AssertionError: Incorrect number of containers on the greedy app expected:<4> but was:<8> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:282) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues(TestFairSchedulerPreemption.java:323) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6240) TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
Sunil G created YARN-6240: - Summary: TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly Key: YARN-6240 URL: https://issues.apache.org/jira/browse/YARN-6240 Project: Hadoop YARN Issue Type: Bug Components: test Reporter: Sunil G *Error Message* Expected to NOT throw exception when refresh queue tries to delete a queue WITHOUT running apps Link [here|https://builds.apache.org/job/PreCommit-YARN-Build/15092/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/] *Stacktrace* {code} java.lang.AssertionError: Expected to NOT throw exception when refresh queue tries to delete a queue WITHOUT running apps at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3875) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4516) [YARN-3368] Use em-table to better render tables
[ https://issues.apache.org/jira/browse/YARN-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-4516. --- Resolution: Won't Fix YARN-5335 already handled this part. Now all tables in new UI is using em-table. > [YARN-3368] Use em-table to better render tables > > > Key: YARN-4516 > URL: https://issues.apache.org/jira/browse/YARN-4516 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Li Lu > > Currently we're using DataTables, it isn't integrated to Ember.js very well. > Instead we can use em-table (see https://github.com/sreenaths/em-table/wiki, > which is created for Tez UI). It supports features such as selectable > columns, pagination, etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5493) In leaf queue page, list applications should only show applications from that leaf queues
[ https://issues.apache.org/jira/browse/YARN-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-5493. --- Resolution: Duplicate Duplicated to YARN-5335. This was fixed while em-table conversion is done. > In leaf queue page, list applications should only show applications from that > leaf queues > - > > Key: YARN-5493 > URL: https://issues.apache.org/jira/browse/YARN-5493 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Yesha Vora > > Steps to reproduce: > * Create a 2 queues > * Go to leaf queue page at http://:/#/yarn-queue-apps/ > url > * click on application list. > Here, it list down all the applications. Instead , It should list down only > applications from that particular leaf queue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6278) -Pyarn-ui build seems broken in trunk
Sunil G created YARN-6278: - Summary: -Pyarn-ui build seems broken in trunk Key: YARN-6278 URL: https://issues.apache.org/jira/browse/YARN-6278 Project: Hadoop YARN Issue Type: Bug Reporter: Sunil G Priority: Critical Link to the error is [here|https://builds.apache.org/job/PreCommit-HDFS-Build/18535/artifact/patchprocess/patch-compile-root.txt] {code} qunit-notifications#0.1.1 bower_components/qunit-notifications [INFO] [INFO] --- exec-maven-plugin:1.3.1:exec (ember build) @ hadoop-yarn-ui --- /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/merge-trees/index.js:33 class MergeTrees { ^ Unexpected reserved word SyntaxError: Unexpected reserved word at Module._compile (module.js:439:25) at Object.Module._extensions..js (module.js:474:10) at Module.load (module.js:356:32) at Function.Module._load (module.js:312:12) at Module.require (module.js:364:17) at require (module.js:380:17) at Object. (/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/broccoli-merge-trees/index.js:2:18) at Module._compile (module.js:456:26) at Object.Module._extensions..js (module.js:474:10) at Module.load (module.js:356:32) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6286) TestCapacityScheduler.testAMLimitUsage throws UndeclaredThrowableException
Sunil G created YARN-6286: - Summary: TestCapacityScheduler.testAMLimitUsage throws UndeclaredThrowableException Key: YARN-6286 URL: https://issues.apache.org/jira/browse/YARN-6286 Project: Hadoop YARN Issue Type: Bug Components: test Reporter: Sunil G {code} testAMLimitUsage(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler) Time elapsed: 0.124 sec <<< ERROR! java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:253) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:218) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:189) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:497) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:384) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:295) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:664) at org.apache.hadoop.yarn.server.resourcemanager.MockRM$2.run(MockRM.java:752) at org.apache.hadoop.yarn.server.resourcemanager.MockRM$2.run(MockRM.java:746) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965) at org.apache.hadoop.yarn.server.resourcemanager.MockRM.submitApp(MockRM.java:765) at org.apache.hadoop.yarn.server.resourcemanager.MockRM.submitApp(MockRM.java:665) at org.apache.hadoop.yarn.server.resourcemanager.MockRM.submitApp(MockRM.java:572) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.verifyAMLimitForLeafQueue(TestCapacityScheduler.java:3370) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testAMLimitUsage(TestCapacityScheduler.java:3232) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5496) Make Node Heatmap Chart categories clickable in new YARN UI
[ https://issues.apache.org/jira/browse/YARN-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-5496. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3 Committed to trunk. Thanks [~GergelyNovak] for the contribution and thanks [~akhilpb] for additional review. > Make Node Heatmap Chart categories clickable in new YARN UI > --- > > Key: YARN-5496 > URL: https://issues.apache.org/jira/browse/YARN-5496 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Gergely Novák > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5496.001.patch, YARN-5496.002.patch, > YARN-5496.003.patch, YARN-5496.004.patch > > > Make Node Heatmap Chart categories clickable. > This Heatmap chart has few categories like 10% used, 30% used etc. > This tags should be clickable. If user clicks on 10% used tag, it should show > hosts with 10% usage. This can be a useful feature for clusters having 1000s > of nodes. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6398) Implement a new native-service UI
Sunil G created YARN-6398: - Summary: Implement a new native-service UI Key: YARN-6398 URL: https://issues.apache.org/jira/browse/YARN-6398 Project: Hadoop YARN Issue Type: Sub-task Components: yarn-native-services Reporter: Sunil G Assignee: Akhil PB Create a new and advanced native service UI which can co-exist with the new Yarn UI. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6402) Move 'Long Running Services' to an independent tab at top level for new Yarn UI
Sunil G created YARN-6402: - Summary: Move 'Long Running Services' to an independent tab at top level for new Yarn UI Key: YARN-6402 URL: https://issues.apache.org/jira/browse/YARN-6402 Project: Hadoop YARN Issue Type: Sub-task Components: webapp Reporter: Sunil G Assignee: Sunil G Currently 'Long Running Services' is a sub link in 'Applications' page. Services could be made top level -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6471) Support to add min/max resource configuration for a queue
Sunil G created YARN-6471: - Summary: Support to add min/max resource configuration for a queue Key: YARN-6471 URL: https://issues.apache.org/jira/browse/YARN-6471 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G This jira will track the new configurations which are needed to configure min resource and max resource of various resource types in a queue. For eg: {noformat} yarn.scheduler.capacity.root.default.memory.min-resource yarn.scheduler.capacity.root.default.memory.max-resource yarn.scheduler.capacity.root.default.vcores.min-resource yarn.scheduler.capacity.root.default.vcores.max-resource {noformat} Uploading a patch soon -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6502) branch-2 build is broken
Sunil G created YARN-6502: - Summary: branch-2 build is broken Key: YARN-6502 URL: https://issues.apache.org/jira/browse/YARN-6502 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Sunil G Priority: Blocker {noformat} [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /home/sunilg/hadoop/repo/branch-2/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java:[183,18] cannot find symbol symbol: method putIfAbsent(org.apache.hadoop.yarn.api.records.ApplicationAttemptId,org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt) location: variable appIdToAppMap of type java.util.Map [ERROR] /home/sunilg/hadoop/repo/branch-2/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java:[184,29] cannot find symbol symbol: method putIfAbsent(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt,org.apache.hadoop.yarn.api.records.Resource) location: variable resourcesPreemptedForApp of type java.util.Map {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6502) branch-2 build is broken
[ https://issues.apache.org/jira/browse/YARN-6502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-6502. --- Resolution: Duplicate Thanks. Closing this as dup. > branch-2 build is broken > > > Key: YARN-6502 > URL: https://issues.apache.org/jira/browse/YARN-6502 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Sunil G >Priority: Blocker > > {code} > [ERROR] COMPILATION ERROR : > [INFO] - > [ERROR] > /home/sunilg/hadoop/repo/branch-2/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java:[183,18] > cannot find symbol > symbol: method > putIfAbsent(org.apache.hadoop.yarn.api.records.ApplicationAttemptId,org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt) > location: variable appIdToAppMap of type > java.util.Map > [ERROR] > /home/sunilg/hadoop/repo/branch-2/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java:[184,29] > cannot find symbol > symbol: method > putIfAbsent(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt,org.apache.hadoop.yarn.api.records.Resource) > location: variable resourcesPreemptedForApp of type > java.util.Map > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6538) Inter Queue preemption is not happening when DRF is configured
Sunil G created YARN-6538: - Summary: Inter Queue preemption is not happening when DRF is configured Key: YARN-6538 URL: https://issues.apache.org/jira/browse/YARN-6538 Project: Hadoop YARN Issue Type: Bug Components: scheduler preemption Affects Versions: 2.8.0 Reporter: Sunil G Assignee: Sunil G Cluster capacity of . Here memory is more and vcores are less. If applications have more demand, vcores might be exhausted. Inter queue preemption ideally has to be kicked in once vcores is over utilized. However preemption is not happening. Analysis: In {{AbstractPreemptableResourceCalculator.computeFixpointAllocation}}, {code} // assign all cluster resources until no more demand, or no resources are // left while (!orderedByNeed.isEmpty() && Resources.greaterThan(rc, totGuarant, unassigned, Resources.none())) { {code} will loop even when vcores are 0. Hence we are having more vcores in idealAssigned which cause no-preemption cases. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6627) Use deployed webapp folder to launch new YARN UI
Sunil G created YARN-6627: - Summary: Use deployed webapp folder to launch new YARN UI Key: YARN-6627 URL: https://issues.apache.org/jira/browse/YARN-6627 Project: Hadoop YARN Issue Type: Sub-task Components: yarn-ui-v2 Reporter: Sunil G Assignee: Sunil G Currently new ui war file is placed in share/hadoop/yarn folder. Along with this, its better to have ui2 folder placed under share/hadoop/yarn/webapps so that UI could be launched from a defined folder. This will also help to make some cutom config related to ui as well. cc/[~jianhe] -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6663) Backport YARN-5333 to branch-2.8, branch-2.7
[ https://issues.apache.org/jira/browse/YARN-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-6663. --- Resolution: Done > Backport YARN-5333 to branch-2.8, branch-2.7 > > > Key: YARN-6663 > URL: https://issues.apache.org/jira/browse/YARN-6663 > Project: Hadoop YARN > Issue Type: Task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5333-branch-2.7.0001.patch, > YARN-5333-branch-2.8.0001.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6747) TestFSAppStarvation.testPreemptionEnable fails intermittently
Sunil G created YARN-6747: - Summary: TestFSAppStarvation.testPreemptionEnable fails intermittently Key: YARN-6747 URL: https://issues.apache.org/jira/browse/YARN-6747 Project: Hadoop YARN Issue Type: Bug Reporter: Sunil G *Error Message* Apps re-added even before starvation delay passed expected:<4> but was:<3> *Stacktrace* java.lang.AssertionError: Apps re-added even before starvation delay passed expected:<4> but was:<3> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.testPreemptionEnabled(TestFSAppStarvation.java:117) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6788) Improve performance of resource profile branch
Sunil G created YARN-6788: - Summary: Improve performance of resource profile branch Key: YARN-6788 URL: https://issues.apache.org/jira/browse/YARN-6788 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G Assignee: Sunil G Currently we could see a 15% performance delta with this branch. Few performance improvements to improve the same. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6789) new api to get all supported resources from RM
Sunil G created YARN-6789: - Summary: new api to get all supported resources from RM Key: YARN-6789 URL: https://issues.apache.org/jira/browse/YARN-6789 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G Assignee: Sunil G It will be better to provide an api to get all supported resource types from RM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6872) Ensure apps could run given NodeLabels are disabled post RM switchover/restart
Sunil G created YARN-6872: - Summary: Ensure apps could run given NodeLabels are disabled post RM switchover/restart Key: YARN-6872 URL: https://issues.apache.org/jira/browse/YARN-6872 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Sunil G Assignee: Sunil G Post YARN-6031, few apps could be failed during recovery provided they had some label requirements for AM and labels were disable post RM restart/switchover. As discussed in YARN-6031, its better to run such apps as it may be long running apps as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6892) Improve API implementation in Resources and DominantResourceCalculator in align to ResourceInformation
Sunil G created YARN-6892: - Summary: Improve API implementation in Resources and DominantResourceCalculator in align to ResourceInformation Key: YARN-6892 URL: https://issues.apache.org/jira/browse/YARN-6892 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G Assignee: Sunil G In YARN-3926, apis in Resources and DRC spents significant cpu cycles in most of its api. For better performance, its better to improve the apis as resource types order is defined in system level (ResourceUtils class ensures this post YARN-6788) This work is preceding to YARN-6788 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6783) DominantResourceCalculator.resourceNames should be a constant
[ https://issues.apache.org/jira/browse/YARN-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-6783. --- Resolution: Duplicate This is no longer needed post YARN-6892 > DominantResourceCalculator.resourceNames should be a constant > - > > Key: YARN-6783 > URL: https://issues.apache.org/jira/browse/YARN-6783 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Priority: Trivial > Labels: newbie > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7013) merge related work for YARN-3926 branch
Sunil G created YARN-7013: - Summary: merge related work for YARN-3926 branch Key: YARN-7013 URL: https://issues.apache.org/jira/browse/YARN-7013 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G Assignee: Sunil G To run jenkins for whole branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7021) TestResourceUtils to be moved to yarn-common
Sunil G created YARN-7021: - Summary: TestResourceUtils to be moved to yarn-common Key: YARN-7021 URL: https://issues.apache.org/jira/browse/YARN-7021 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G ResourceUtils class is now in yarn-api. Its better its test class also to be moved there, however these tests using lot of resources and using ConfigurationProvider which is available only in yarn-common. Hence investigate and improve test for ResourceUtils class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7039) Fix javac and javadoc errors in YARN-3926 branch
Sunil G created YARN-7039: - Summary: Fix javac and javadoc errors in YARN-3926 branch Key: YARN-7039 URL: https://issues.apache.org/jira/browse/YARN-7039 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: YARN-3926 Reporter: Sunil G Assignee: Sunil G Fix javac and doc errors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6782) ResourceUtils.getResourceTypes() reloads the resource configuration every time
[ https://issues.apache.org/jira/browse/YARN-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-6782. --- Resolution: Done > ResourceUtils.getResourceTypes() reloads the resource configuration every time > -- > > Key: YARN-6782 > URL: https://issues.apache.org/jira/browse/YARN-6782 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Priority: Minor > Labels: newbie++ > > Many objects need to collect the list of configured resources upon creation. > The only way do that now is by calling {{getResourceTypes()}}, which reloads > the config. The net result is that the config is loaded and reloaded over > and over again during startup. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7056) Document Resource Profiles feature
Sunil G created YARN-7056: - Summary: Document Resource Profiles feature Key: YARN-7056 URL: https://issues.apache.org/jira/browse/YARN-7056 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7069) [Umbrella] YARN Resource Profiles extended enhancements
Sunil G created YARN-7069: - Summary: [Umbrella] YARN Resource Profiles extended enhancements Key: YARN-7069 URL: https://issues.apache.org/jira/browse/YARN-7069 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager, resourcemanager Reporter: Sunil G This ticket to is to track further enhancements to resource profile work happening in YARN-3926 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7093) Improve log message in ResourceUtils
Sunil G created YARN-7093: - Summary: Improve log message in ResourceUtils Key: YARN-7093 URL: https://issues.apache.org/jira/browse/YARN-7093 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G Assignee: Sunil G Priority: Trivial Improve log message ResourceUtils class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6839) [YARN-3368] Be able to view application / container logs on the new YARN UI.
[ https://issues.apache.org/jira/browse/YARN-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-6839. --- Resolution: Duplicate > [YARN-3368] Be able to view application / container logs on the new YARN UI. > > > Key: YARN-6839 > URL: https://issues.apache.org/jira/browse/YARN-6839 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan > > Currently viewing application/container logs will be redirected to the old > UI, we should leverage the new UI capabilities to make a better log-viewing > experiences. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7157) Support displaying per-user's apps in RM UI page and in secure cluster
Sunil G created YARN-7157: - Summary: Support displaying per-user's apps in RM UI page and in secure cluster Key: YARN-7157 URL: https://issues.apache.org/jira/browse/YARN-7157 Project: Hadoop YARN Issue Type: Bug Components: webapp Reporter: Sunil G Assignee: Sunil G A user who is accessing a secure cluster via a secure UI should be able to see only his/her own apps. This feature will not break any compatibility as it will turned off by default -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7170) Investigate bower dependencies for YARN UI v2
Sunil G created YARN-7170: - Summary: Investigate bower dependencies for YARN UI v2 Key: YARN-7170 URL: https://issues.apache.org/jira/browse/YARN-7170 Project: Hadoop YARN Issue Type: Bug Components: webapp Reporter: Sunil G Assignee: Sunil G [INFO] bower ember#2.2.0 progress Receiving objects: 50% (38449/75444), 722.46 MiB | 3.30 MiB/s ... [INFO] bower ember#2.2.0 progress Receiving objects: 99% (75017/75444), 1.56 GiB | 3.31 MiB/s Investigate the dependencies and reduce the download size and speed of compilation. cc/ [~Sreenath] and [~akhilpb] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7184) Test Jira
[ https://issues.apache.org/jira/browse/YARN-7184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-7184. --- Resolution: Not A Problem Reopened and closed as "Not a problem" > Test Jira > - > > Key: YARN-7184 > URL: https://issues.apache.org/jira/browse/YARN-7184 > Project: Hadoop YARN > Issue Type: Task > Environment: Test variable is missing in the test class >Reporter: lovekesh bansal > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7254) UI and metrics changes related to absolute resource configuration
Sunil G created YARN-7254: - Summary: UI and metrics changes related to absolute resource configuration Key: YARN-7254 URL: https://issues.apache.org/jira/browse/YARN-7254 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G Impact on UI and metrics related to absolute resource configuration on CS -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7287) Fix typo in ContainerLaunch
[ https://issues.apache.org/jira/browse/YARN-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-7287. --- Resolution: Duplicate Marking as duplicate to YARN-7279 > Fix typo in ContainerLaunch > --- > > Key: YARN-7287 > URL: https://issues.apache.org/jira/browse/YARN-7287 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ajay Kumar >Priority: Trivial > > Fix typo in ContainerLaunch. > {code} > .append(" mapreduce.reduce.e nv\n") > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7314) Multiple Resource manager test cases failing
[ https://issues.apache.org/jira/browse/YARN-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-7314. --- Resolution: Duplicate > Multiple Resource manager test cases failing > > > Key: YARN-7314 > URL: https://issues.apache.org/jira/browse/YARN-7314 > Project: Hadoop YARN > Issue Type: Bug >Reporter: lovekesh bansal >Priority: Critical > > Following 14 unit tests are failing in the trunk: > > org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService.testResourceTypes > > > org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart.testSchedulerRecovery[FAIR] > > > org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart.testDynamicQueueRecovery[FAIR] > > > org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter.testRMWritingMassiveHistoryForFairSche > > > org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter.testRMWritingMassiveHistoryForCapacitySche > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt.testHeadroomWithBlackListedNodes > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueMaxAMShare > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testMultipleCompletedEvent > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testRefreshQueuesWhenRMHA > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueMaxAMShareWithContainerReservation > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testComputeMaxAMResource > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueMaxAMShareDefault > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testReservationMetrics > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testRequestAMResourceInZeroFairShareQueue > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7324) TestCapacitySchedulerWithMultiResourceTypes.testBasicCapacitySchedulerWithProfile fails in trunk
Sunil G created YARN-7324: - Summary: TestCapacitySchedulerWithMultiResourceTypes.testBasicCapacitySchedulerWithProfile fails in trunk Key: YARN-7324 URL: https://issues.apache.org/jira/browse/YARN-7324 Project: Hadoop YARN Issue Type: Bug Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G *Error Message* Attempt state is not correct (timeout). expected: but was: *Stacktrace* {noformat} java.lang.AssertionError: Attempt state is not correct (timeout). expected: but was: at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerWithMultiResourceTypes.testBasicCapacitySchedulerWithProfile(TestCapacitySchedulerWithMultiResourceTypes.java:108) {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7324) TestCapacitySchedulerWithMultiResourceTypes.testBasicCapacitySchedulerWithProfile fails in trunk
[ https://issues.apache.org/jira/browse/YARN-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-7324. --- Resolution: Invalid Marking as invalid as its a branch specific issue. > TestCapacitySchedulerWithMultiResourceTypes.testBasicCapacitySchedulerWithProfile > fails in trunk > > > Key: YARN-7324 > URL: https://issues.apache.org/jira/browse/YARN-7324 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > > *Error Message* > Attempt state is not correct (timeout). expected: but > was: > *Stacktrace* > {noformat} > java.lang.AssertionError: Attempt state is not correct (timeout). > expected: but was: > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerWithMultiResourceTypes.testBasicCapacitySchedulerWithProfile(TestCapacitySchedulerWithMultiResourceTypes.java:108) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7331) Change in few metrics in new YARN UI related to native-services
Sunil G created YARN-7331: - Summary: Change in few metrics in new YARN UI related to native-services Key: YARN-7331 URL: https://issues.apache.org/jira/browse/YARN-7331 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: yarn-native-services Reporter: Sunil G Assignee: Sunil G Few metrics changes. 1. The below metrics need not to show up in UI. These can be removed. containersRequested pendingAAContainers surplusContainers Containers Failed Since Last Threshold Pending Anti-Affinity Containers 2. completedContainers is Renamed to containersSucceeded 3. containersReady is newly added. 4. “Created Time” in component to be removed. This is not real created time for component. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7332) Compute effectiveCapacity per each resource vector
Sunil G created YARN-7332: - Summary: Compute effectiveCapacity per each resource vector Key: YARN-7332 URL: https://issues.apache.org/jira/browse/YARN-7332 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Affects Versions: YARN-5881 Reporter: Sunil G Assignee: Sunil G Currently effective capacity uses a generalized approach based on dominance. Hence some vectors may not be calculated correctly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7364) Queue dash board in new YARN UI has incorrect values
Sunil G created YARN-7364: - Summary: Queue dash board in new YARN UI has incorrect values Key: YARN-7364 URL: https://issues.apache.org/jira/browse/YARN-7364 Project: Hadoop YARN Issue Type: Bug Components: webapp Reporter: Sunil G Assignee: Sunil G Queue dashboard in cluster over page does not show queue metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7410) Cleanup FixedValueResource to avoid dependency to ResourceUtils
Sunil G created YARN-7410: - Summary: Cleanup FixedValueResource to avoid dependency to ResourceUtils Key: YARN-7410 URL: https://issues.apache.org/jira/browse/YARN-7410 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.1.0 Reporter: Sunil G Assignee: Wangda Tan Currently FixedValue Resource constants has some dependencies to ResourceUtils. This jira will help to cleanup this dependencies. Thanks [~leftnoteasy] for finding the same. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7411) Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee
Sunil G created YARN-7411: - Summary: Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee Key: YARN-7411 URL: https://issues.apache.org/jira/browse/YARN-7411 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: YARN-5881 Reporter: Sunil G Assignee: Sunil G {{normalizedGuarantee}} is computed based on queue's capacity. This has to be updated correctly when CS starts to accept queue's capacity in terms of absolute resource. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-1963) Support priorities across applications within the same queue
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-1963. --- Resolution: Fixed Closing this issue as all sub-items are done. > Support priorities across applications within the same queue > - > > Key: YARN-1963 > URL: https://issues.apache.org/jira/browse/YARN-1963 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, resourcemanager >Reporter: Arun C Murthy >Assignee: Sunil G > Attachments: 0001-YARN-1963-prototype.patch, YARN Application > Priorities Design _02.pdf, YARN Application Priorities Design.pdf, YARN > Application Priorities Design_01.pdf > > > It will be very useful to support priorities among applications within the > same queue, particularly in production scenarios. It allows for finer-grained > controls without having to force admins to create a multitude of queues, plus > allows existing applications to continue using existing queues which are > usually part of institutional memory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6627) Use deployed webapp folder to launch new YARN UI
[ https://issues.apache.org/jira/browse/YARN-6627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-6627. --- Resolution: Fixed Fix Version/s: 2.9.0 Thanks [~asuresh] for backporting this to 2.9.0 (ref: d15be3c93a0d03d15eb2407d266680343b51bf15). Closing this ticket by updating fix version. > Use deployed webapp folder to launch new YARN UI > > > Key: YARN-6627 > URL: https://issues.apache.org/jira/browse/YARN-6627 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Sunil G >Assignee: Sunil G > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: YARN-6627.0001.patch, YARN-6627.0002.patch, > YARN-6627.0003.patch, YARN-6627.branch-2.001.patch > > > Currently new ui war file is placed in share/hadoop/yarn folder. Along with > this, its better to have ui2 folder placed under share/hadoop/yarn/webapps so > that UI could be launched from a defined folder. > This will also help to make some cutom config related to ui as well. > cc/[~jianhe] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7482) Max applications calculation per queue has to be retrospected with absolute resource support
Sunil G created YARN-7482: - Summary: Max applications calculation per queue has to be retrospected with absolute resource support Key: YARN-7482 URL: https://issues.apache.org/jira/browse/YARN-7482 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Affects Versions: YARN-5881 Reporter: Sunil G Assignee: Sunil G Retrospect max system applications w.r.t absolute resource configuration in a queue -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7483) Test cases cleanup post YARN-5881
Sunil G created YARN-7483: - Summary: Test cases cleanup post YARN-5881 Key: YARN-7483 URL: https://issues.apache.org/jira/browse/YARN-7483 Project: Hadoop YARN Issue Type: Sub-task Components: test Affects Versions: YARN-5881 Reporter: Sunil G Assignee: Sunil G Few UT cases has to be cleaned or rewritten cleanly to consider both absolute/percentage based configuration. Related test classes # TestLeafQueue # TestCapacityScheduler -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7494) Add muti node lookup support for better placement
Sunil G created YARN-7494: - Summary: Add muti node lookup support for better placement Key: YARN-7494 URL: https://issues.apache.org/jira/browse/YARN-7494 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G Instead of single node, for effectiveness we can consider a multi node lookup based on partition to start with. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7510) Merge work for YARN-5881
Sunil G created YARN-7510: - Summary: Merge work for YARN-5881 Key: YARN-7510 URL: https://issues.apache.org/jira/browse/YARN-7510 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G Merge YARN-5881 work -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7533) Documentation for absolute resource support in CS
Sunil G created YARN-7533: - Summary: Documentation for absolute resource support in CS Key: YARN-7533 URL: https://issues.apache.org/jira/browse/YARN-7533 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7538) Performance improvement related to absolute min-max resources
Sunil G created YARN-7538: - Summary: Performance improvement related to absolute min-max resources Key: YARN-7538 URL: https://issues.apache.org/jira/browse/YARN-7538 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G performance improvements. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7544) Configuration cleanup associated to support absolute resources in a queuec
Sunil G created YARN-7544: - Summary: Configuration cleanup associated to support absolute resources in a queuec Key: YARN-7544 URL: https://issues.apache.org/jira/browse/YARN-7544 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G Its better to reuse existing *capacity* and *max-capacity* instead of introducing new config items. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7564) Fix checkstyle issues of YARN-5881 branch
Sunil G created YARN-7564: - Summary: Fix checkstyle issues of YARN-5881 branch Key: YARN-7564 URL: https://issues.apache.org/jira/browse/YARN-7564 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G Assignee: Sunil G Priority: Minor fix jenkins issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7573) Gpu Information page could be empty for nodes without GPU
Sunil G created YARN-7573: - Summary: Gpu Information page could be empty for nodes without GPU Key: YARN-7573 URL: https://issues.apache.org/jira/browse/YARN-7573 Project: Hadoop YARN Issue Type: Sub-task Components: webapp, yarn-ui-v2 Reporter: Sunil G Assignee: Sunil G In new YARN UI, node page is not accessible if that node doesnt have any GPU. Also Under node page, when we click on "List of Containers/Applications", Gpu Information left nave is disappearing. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7092) Render application specific log under application tab in new YARN UI
[ https://issues.apache.org/jira/browse/YARN-7092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-7092. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Thanks [~akhilpb] for working on this patch and thanks [~skmvasu] for helping in rebasing and review. Committed to trunk > Render application specific log under application tab in new YARN UI > > > Key: YARN-7092 > URL: https://issues.apache.org/jira/browse/YARN-7092 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Akhil PB >Assignee: Akhil PB > Fix For: 3.1.0 > > Attachments: YARN-7092.001.patch > > > Feature to view application logs in new yarn-ui. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7635) TestRMWebServicesSchedulerActivities fails in trunk
Sunil G created YARN-7635: - Summary: TestRMWebServicesSchedulerActivities fails in trunk Key: YARN-7635 URL: https://issues.apache.org/jira/browse/YARN-7635 Project: Hadoop YARN Issue Type: Bug Components: test Affects Versions: 3.1.0 Reporter: Sunil G Error trace. {code} [ERROR] Tests run: 18, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 41.297 s <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities [ERROR] testClusterScheduler(org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities) Time elapsed: 0.435 s <<< FAILURE! java.lang.AssertionError: incorrect number of elements expected:<33> but was:<34> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched.verifySubQueue(TestRMWebServicesCapacitySched.java:363) at org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched.verifySubQueue(TestRMWebServicesCapacitySched.java:384) at org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched.verifyClusterScheduler(TestRMWebServicesCapacitySched.java:340) at org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched.testClusterScheduler(TestRMWebServicesCapacitySched.java:182) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7637) GPU volume creation command fails when work preserving is disabled at NM
Sunil G created YARN-7637: - Summary: GPU volume creation command fails when work preserving is disabled at NM Key: YARN-7637 URL: https://issues.apache.org/jira/browse/YARN-7637 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 3.1.0 Reporter: Sunil G Assignee: Zian Chen When work preserving is disabled, NM uses {{NMNullStateStoreService}}. Hence resource mappings related to GPU wont be saved at Container. This has to be rechecked and store accordingly. cc/ [~leftnoteasy] and [~Zian Chen] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7640) Retrospect to enable CORS related configs by default in YARN
Sunil G created YARN-7640: - Summary: Retrospect to enable CORS related configs by default in YARN Key: YARN-7640 URL: https://issues.apache.org/jira/browse/YARN-7640 Project: Hadoop YARN Issue Type: Bug Components: nodemanager, resourcemanager Affects Versions: 3.0.0-beta1 Reporter: Sunil G Currently admin has to do below config changes to enable CORS in YARN. {code} 1. Add org.apache.hadoop.security.HttpCrossOriginFilterInitializer to hadoop.http.filter.initializers 2. Set hadoop.http.cross-origin.enabled to true 3. Set hadoop.http.cross-origin.allowed-methods to GET,HEAD 4. Set yarn.nodemanager.webapp.cross-origin.enabled to true 5. Set yarn.resourcemanager.webapp.cross-origin.enabled to true {code} For better handling, we could enable this config by default in YARN. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7650) Queue link in old RM UI is not working correctly when node labels are enabled
Sunil G created YARN-7650: - Summary: Queue link in old RM UI is not working correctly when node labels are enabled Key: YARN-7650 URL: https://issues.apache.org/jira/browse/YARN-7650 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.8.0 Reporter: Sunil G When Queue link is clicked from application page, {{http://hw12715:8088/cluster/app/application_1513168152296_0001}}, we are getting below URL {code} http://hw12715:8088/cluster/scheduler?openQueues=a1 {code} It should be like below when node labels are enabled else Queue wont be opened up. {code} http://localhost:8088/cluster/scheduler?openQueues=a1#Partition: x #Queue: root#Queue: a#Queue: a.a1 {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7651) branch-2 application master (MR) cannot run in 3.1 cluster
Sunil G created YARN-7651: - Summary: branch-2 application master (MR) cannot run in 3.1 cluster Key: YARN-7651 URL: https://issues.apache.org/jira/browse/YARN-7651 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.1.0 Reporter: Sunil G Priority: Blocker {noformat} 2017-12-13 19:21:20,452 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-12-13 19:21:20,481 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.RuntimeException: Unable to determine current user at org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:253) at org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:219) at org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:211) at org.apache.hadoop.conf.Configuration.addResource(Configuration.java:876) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1571) Caused by: java.io.IOException: Exception reading /Users/sunilgovindan/install/hadoop/tmp/nm-local-dir/usercache/sunilgovindan/appcache/application_1513172966925_0001/container_1513172966925_0001_01_01/container_tokens at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:870) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:803) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:676) at org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:251) ... 4 more Caused by: java.io.IOException: Unknown version 1 in token storage. at org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226) at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205) ... 8 more 2017-12-13 19:21:20,484 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1 {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7666) Introduce scheduler specific environment variable support in ASC for better scheduling placement configurations
Sunil G created YARN-7666: - Summary: Introduce scheduler specific environment variable support in ASC for better scheduling placement configurations Key: YARN-7666 URL: https://issues.apache.org/jira/browse/YARN-7666 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G Assignee: Sunil G Introduce a scheduler specific key-value map to hold env variables in ASC. And also convert AppPlacementAllocator initialization to each app based on policy configured at each app. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-7684. --- Resolution: Won't Fix This is already covered in 2.8+ version. YARN-4484 could be backported to 2.7 as needed. > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Fix For: 2.8.0 > > Attachments: TestResultsAfterPortingYARN-4484.pdf, > YARN-7684-branch-2.7.3.001.patch, yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7699) queueUsagePercentage is coming as INF for getApp REST api call
Sunil G created YARN-7699: - Summary: queueUsagePercentage is coming as INF for getApp REST api call Key: YARN-7699 URL: https://issues.apache.org/jira/browse/YARN-7699 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 3.0.0, 2.9.0 Reporter: Sunil G Assignee: Sunil G REST o/p for getApp call where queueUsagePercentage is INF. {noformat} { "app": { "id": "application_1514964945154_0001", "user": "hrt_qa", "name": "DistributedShell", "queue": "a1", "state": "RUNNING", "finalStatus": "UNDEFINED", "progress": 90.0, "trackingUI": "ApplicationMaster", "trackingUrl": "http://ctr-e136-1513029738776-28101-01-04.hwx.site:8088/proxy/application_1514964945154_0001/";, "diagnostics": "", "clusterId": 1514964945154, "applicationType": "YARN", "applicationTags": "", "priority": 0, "startedTime": 1514965015754, "finishedTime": 0, "elapsedTime": 418844, "amContainerLogs": "http://ctr-e136-1513029738776-28101-01-07.hwx.site:8042/node/containerlogs/container_e43_1514964945154_0001_01_01/hrt_qa";, "amHostHttpAddress": "ctr-e136-1513029738776-28101-01-07.hwx.site:8042", "allocatedMB": 600, "allocatedVCores": 1, "reservedMB": 0, "reservedVCores": 0, "runningContainers": 1, "memorySeconds": 1227070, "vcoreSeconds": 2041, "queueUsagePercentage": INF, "clusterUsagePercentage": 3.2552083, "resourceSecondsMap": { "entry": { "key": "memory-mb", "value": "1227070" }, "entry": { "key": "vcores", "value": "2041" } }, "preemptedResourceMB": 0, "preemptedResourceVCores": 0, "numNonAMContainerPreempted": 0, "numAMContainerPreempted": 0, "preemptedMemorySeconds": 0, "preemptedVcoreSeconds": 0, "preemptedResourceSecondsMap": { }, "resourceRequests": [ { "priority": 0, "resourceName": "*", "capability": { "memory": 600, "vCores": 1 }, "numContainers": 1, "relaxLocality": true, "nodeLabelExpression": "x", "executionTypeRequest": { "executionType": "GUARANTEED", "enforceExecutionType": false }, "enforceExecutionType": false } ], "logAggregationStatus": "NOT_START", "unmanagedApplication": false, "appNodeLabelExpression": "x", "amNodeLabelExpression": "x", "resourceInfo": { "resourceUsagesByPartition": [ { "partitionName": "", "used": { "memory": 0, "vCores": 0 }, "reserved": { "memory": 0, "vCores": 0 }, "pending": { "memory": 0, "vCores": 0 }, "amUsed": { "memory": 0, "vCores": 0 }, "amLimit": { "memory": 0, "vCores": 0 } }, { "partitionName": "x", "used": { "memory": 0, "vCores": 0 }, "reserved": { "memory": 0, "vCores": 0 }, "pending": { "memory": 600, "vCores": 1 }, "amUsed": { "memory": 0, "vCores": 0 }, "amLimit": { "memory": 0, "vCores": 0 } }, { "partitionName": "y", "used": { "memory": 600, "vCores": 1 }, "reserved": { "memory": 0, "vCores": 0 }, "pending": { "memory": 0, "vCores": 0 }, "amUsed": { "mem
[jira] [Created] (YARN-7753) [UI2] Application logs has to be pulled from ATS 1.5 instead of ATS2
Sunil G created YARN-7753: - Summary: [UI2] Application logs has to be pulled from ATS 1.5 instead of ATS2 Key: YARN-7753 URL: https://issues.apache.org/jira/browse/YARN-7753 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Reporter: Sunil G Assignee: Sunil G Currently UI tries to pull logs from ATS v2. Instead, it should be pulled from ATS v1 as ATS2 doesnt have a log story yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7792) Merge work for YARN-6592
Sunil G created YARN-7792: - Summary: Merge work for YARN-6592 Key: YARN-7792 URL: https://issues.apache.org/jira/browse/YARN-7792 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G This Jira is to run aggregated YARN-6592 branch patch against trunk and check for any jenkins issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7794) SLSRunner is not loading timeline service jars causing failure
Sunil G created YARN-7794: - Summary: SLSRunner is not loading timeline service jars causing failure Key: YARN-7794 URL: https://issues.apache.org/jira/browse/YARN-7794 Project: Hadoop YARN Issue Type: Bug Components: scheduler-load-simulator Affects Versions: 3.1.0 Reporter: Sunil G {code:java} Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 13 more Exception in thread "pool-2-thread-390" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollector at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:443) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:321) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:641){code} We are getting this error while running SLS. new patch of timelineservice under share/hadoop/yarn is not loaded in SLS jvm (verified from slsrunner classpath) cc/ [~rohithsharma] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7795) Fix jenkins issues of YARN-6592 branch
Sunil G created YARN-7795: - Summary: Fix jenkins issues of YARN-6592 branch Key: YARN-7795 URL: https://issues.apache.org/jira/browse/YARN-7795 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G Assignee: Sunil G Refer link . Also fix the javadoc errors as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7861) [UI2] Logs page shows duplicated containers with ATS
Sunil G created YARN-7861: - Summary: [UI2] Logs page shows duplicated containers with ATS Key: YARN-7861 URL: https://issues.apache.org/jira/browse/YARN-7861 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Environment: There were couple of issues: # duplicated container listed from RM and ATS in log container list # log page has to be cleared every time same page is accessed Reporter: Sunil G Assignee: Sunil G -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7862) YARN native service REST endpoint needs user.name as query param
Sunil G created YARN-7862: - Summary: YARN native service REST endpoint needs user.name as query param Key: YARN-7862 URL: https://issues.apache.org/jira/browse/YARN-7862 Project: Hadoop YARN Issue Type: Bug Components: yarn-native-services Reporter: Sunil G While accessing below yarn rest end point with POST method type, {code:java} http://rm_ip:8088/app/v1/services{code} below error is coming in non-secure cluster. {noformat} { "diagnostics": "Null user" }{noformat} When *user.name* is provided as query param with *dr.who* we can see that yarn started service with proxy user, not dr.who. In non-secure cluster, native service should ideally take the user from remote ugi. in secure cluster, its better to derive user from kerberized shell. cc/ [~jianhe] [~eyang] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7863) Modify placement constraints to support node attributes
Sunil G created YARN-7863: - Summary: Modify placement constraints to support node attributes Key: YARN-7863 URL: https://issues.apache.org/jira/browse/YARN-7863 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G Assignee: Sunil G This Jira will track to *Modify existing placement constraints to support node attributes.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7802) [UI2] Application regex search did not work properly with app name
[ https://issues.apache.org/jira/browse/YARN-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-7802. --- Resolution: Fixed Hadoop Flags: Reviewed Thanks [~Sreenath] > [UI2] Application regex search did not work properly with app name > -- > > Key: YARN-7802 > URL: https://issues.apache.org/jira/browse/YARN-7802 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Yesha Vora >Assignee: Sreenath Somarajapuram >Priority: Major > Attachments: YARN-7802.1.patch, YARN-7802.2.patch > > > Steps: > 1) Start yarn services with "yesha-hbase-retry-2" > 2) put regex = yesha-hbase-retry-2 > http://host:8088/ui2/#/yarn-apps/apps?searchText=yesha-hbase-retry-2 > Here, the application does not gets listed. The regex work with > "yesha-hbase-retry-" input but does not work with full app name. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7866) [UI2] Kerberizing the UI doesn't give any warning or content when UI is accessed without kinit
Sunil G created YARN-7866: - Summary: [UI2] Kerberizing the UI doesn't give any warning or content when UI is accessed without kinit Key: YARN-7866 URL: https://issues.apache.org/jira/browse/YARN-7866 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Reporter: Sunil G Assignee: Sunil G Handle 401 error and show in UI -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7832) Logs page does not work for Running applications
[ https://issues.apache.org/jira/browse/YARN-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-7832. --- Resolution: Not A Problem Thanks [~yeshavora] for confirming, This is working fine with Combine System Metric Publisher mode > Logs page does not work for Running applications > > > Key: YARN-7832 > URL: https://issues.apache.org/jira/browse/YARN-7832 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.0.0 >Reporter: Yesha Vora >Assignee: Sunil G >Priority: Critical > Attachments: Screen Shot 2018-01-26 at 3.28.40 PM.png, > YARN-7832.001.patch > > > Scenario > * Run yarn service application > * When application is Running, go to log page > * Select AttemptId and Container Id > Logs are not showed on UI. It complains "No log data available!" > > Here > [http://xxx:8188/ws/v1/applicationhistory/containers/container_e07_1516919074719_0004_01_01/logs?_=1517009230358] > API fails with 500 Internal Server Error. > {"exception":"WebApplicationException","message":"java.io.IOException: > ","javaClassName":"javax.ws.rs.WebApplicationException"} > {code:java} > GET > http://xxx:8188/ws/v1/applicationhistory/containers/container_e07_1516919074719_0004_01_01/logs?_=1517009230358 > 500 (Internal Server Error) > (anonymous) @ VM779:1 > send @ vendor.js:572 > ajax @ vendor.js:548 > (anonymous) @ vendor.js:5119 > initializePromise @ vendor.js:2941 > Promise @ vendor.js:3005 > ajax @ vendor.js:5117 > ajax @ yarn-ui.js:1 > superWrapper @ vendor.js:1591 > query @ vendor.js:5112 > ember$data$lib$system$store$finders$$_query @ vendor.js:5177 > query @ vendor.js:5334 > fetchLogFilesForContainerId @ yarn-ui.js:132 > showLogFilesForContainerId @ yarn-ui.js:126 > run @ vendor.js:648 > join @ vendor.js:648 > run.join @ vendor.js:1510 > closureAction @ vendor.js:1865 > trigger @ vendor.js:302 > (anonymous) @ vendor.js:339 > each @ vendor.js:61 > each @ vendor.js:51 > trigger @ vendor.js:339 > d.select @ vendor.js:5598 > (anonymous) @ vendor.js:5598 > d.invoke @ vendor.js:5598 > d.trigger @ vendor.js:5598 > e.trigger @ vendor.js:5598 > (anonymous) @ vendor.js:5598 > d.invoke @ vendor.js:5598 > d.trigger @ vendor.js:5598 > (anonymous) @ vendor.js:5598 > dispatch @ vendor.js:306 > elemData.handle @ vendor.js:281{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7770) Support for setting application priority for Distributed Shell jobs
[ https://issues.apache.org/jira/browse/YARN-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-7770. --- Resolution: Not A Bug > Support for setting application priority for Distributed Shell jobs > --- > > Key: YARN-7770 > URL: https://issues.apache.org/jira/browse/YARN-7770 > Project: Hadoop YARN > Issue Type: Bug > Components: applications/distributed-shell >Reporter: Charan Hebri >Assignee: Sunil G >Priority: Major > > Currently there isn't a way to submit a Distributed Shell job with an > application priority like how it is done via the property > {noformat} > mapred.job.priority{noformat} > for MapReduce jobs. Creating this issue to track support for the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7912) While launching Native Service app from UI, consider service owner name from user.name query parameter
Sunil G created YARN-7912: - Summary: While launching Native Service app from UI, consider service owner name from user.name query parameter Key: YARN-7912 URL: https://issues.apache.org/jira/browse/YARN-7912 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Reporter: Sunil G As per comments from [~eyang] in YARN-7827, "For supporting knox, it would be good for javascript to detect the url entering /ui2 and process [user.name|http://user.name/] property. If there isn't one found, then proceed with ajax call to resource manager to find out who is the current user to pass the parameter along the rest api calls." This Jira will track to handle this. This is now pending feasibility check. Thanks [~eyang] and [~jianhe] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7993) [UI2] yarn-service page need to consider ServiceState to show stop/delete buttons
Sunil G created YARN-7993: - Summary: [UI2] yarn-service page need to consider ServiceState to show stop/delete buttons Key: YARN-7993 URL: https://issues.apache.org/jira/browse/YARN-7993 Project: Hadoop YARN Issue Type: Sub-task Components: yarn-ui-v2 Reporter: Sunil G yarn service page has stop/delete buttons. These buttons has to be shown/hidden based on ServiceState of each app from ATS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8040) [UI2] yarn new ui web-app does not respect current pathname for REST api
Sunil G created YARN-8040: - Summary: [UI2] yarn new ui web-app does not respect current pathname for REST api Key: YARN-8040 URL: https://issues.apache.org/jira/browse/YARN-8040 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Reporter: Sunil G Assignee: Sunil G When ui2 is accessed behind proxy like knox/nginx, trailing path name should not be skipped. However trim of "ui2" if its there. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8068) Upgrading apps from Hadoop 2.7 based clients to 2.8+ cause NPE in app timeline publish
Sunil G created YARN-8068: - Summary: Upgrading apps from Hadoop 2.7 based clients to 2.8+ cause NPE in app timeline publish Key: YARN-8068 URL: https://issues.apache.org/jira/browse/YARN-8068 Project: Hadoop YARN Issue Type: Bug Components: yarn Affects Versions: 2.8.3 Reporter: Sunil G Assignee: Sunil G [TimelineServiceV1Publisher|eclipse-javadoc:%E2%98%82=hadoop-yarn-server-resourcemanager/src%5C/main%5C/java%3Corg.apache.hadoop.yarn.server.resourcemanager.metrics%7BTimelineServiceV1Publisher.java%E2%98%83TimelineServiceV1Publisher].appCreated will cause NPE as we use like below {code:java} entityInfo.put(ApplicationMetricsConstants.APPLICATION_PRIORITY_INFO, app.getApplicationPriority().getPriority());{code} We have to handle this case while recovery. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8115) [UI2] URL data like nodeHTTPAddress must be encoded in UI before using for NM/ATS
Sunil G created YARN-8115: - Summary: [UI2] URL data like nodeHTTPAddress must be encoded in UI before using for NM/ATS Key: YARN-8115 URL: https://issues.apache.org/jira/browse/YARN-8115 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Reporter: Sunil G Assignee: Sreenath Somarajapuram nodeHTTPAddress is picked from RM ,but it should be encoded before processing it from UI -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org