[jira] [Comment Edited] (YARN-7858) Support special Node Attribute scopes in addition to NODE and RACK
[
https://issues.apache.org/jira/browse/YARN-7858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358050#comment-16358050
]
Weiwei Yang edited comment on YARN-7858 at 2/9/18 7:53 AM:
---
Hi [~asuresh]
Could you please elaborate what do you mean by "to allow ALL attributes to be
scope-able or just a pre-configured subset"? I took a look at this today, I
think in terms of node-attributes, the only scope makes sense is "*NODE*". E.g
{code}
{target: node-attribute:host IN "n1,n2,n3", scope: "host"}
{code}
that constrains the allocations to nodes where attribute {{host}} is "n1", "n2"
or "n3". You can't specify an attribute in scope right? Just want to get to the
same page.
Thanks
was (Author: cheersyang):
Hi [~asuresh]
Could you please elaborate what do you mean by "to allow ALL attributes to be
scope-able or just a pre-configured subset"? I took a look at this today, I
think in terms of node-attributes, the only scope makes sense is "*NODE*". E.g
{target: node-attribute:*host* IN "n1,n2,n3", *scope*: "host"}
that constrains the allocations to nodes where attribute {{host}} is "n1", "n2"
or "n3". You can't specify an attribute in scope right? Just want to get to the
same page.
Thanks
> Support special Node Attribute scopes in addition to NODE and RACK
> --
>
> Key: YARN-7858
> URL: https://issues.apache.org/jira/browse/YARN-7858
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Weiwei Yang
>Priority: Major
>
> Currently, we have only two scopes defined: NODE and RACK against which we
> check the cardinality of the placement.
> This idea should be extended to support node-attribute scopes. For eg:
> Placement of containers across *upgrade domains* and *failure domains*.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7858) Support special Node Attribute scopes in addition to NODE and RACK
[
https://issues.apache.org/jira/browse/YARN-7858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358050#comment-16358050
]
Weiwei Yang commented on YARN-7858:
---
Hi [~asuresh]
Could you please elaborate what do you mean by "to allow ALL attributes to be
scope-able or just a pre-configured subset"? I took a look at this today, I
think in terms of node-attributes, the only scope makes sense is "*NODE*". E.g
{target: node-attribute:*host* IN "n1,n2,n3", *scope*: "host"}
that constrains the allocations to nodes where attribute {{host}} is "n1", "n2"
or "n3". You can't specify an attribute in scope right? Just want to get to the
same page.
Thanks
> Support special Node Attribute scopes in addition to NODE and RACK
> --
>
> Key: YARN-7858
> URL: https://issues.apache.org/jira/browse/YARN-7858
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Weiwei Yang
>Priority: Major
>
> Currently, we have only two scopes defined: NODE and RACK against which we
> check the cardinality of the placement.
> This idea should be extended to support node-attribute scopes. For eg:
> Placement of containers across *upgrade domains* and *failure domains*.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6462) Add yarn command to list all queues
[ https://issues.apache.org/jira/browse/YARN-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shen Yinjie updated YARN-6462: -- Description: we need a yarn command to list all leaf queues. especially in large scale cluster ,there are a large amount of queues in tree-format for various apps , we actually need a list-all-leaf-queues interface to get queues infomation immediately ,other than search in fair scheduler.xml or in yarn-scheduler web layer by layer.sometimes we should also vertify a new queue is successfully added in scheduer, instead of fail for some format error or other wise. was:we need a yarn command to list all leaf queues. > Add yarn command to list all queues > --- > > Key: YARN-6462 > URL: https://issues.apache.org/jira/browse/YARN-6462 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Shen Yinjie >Assignee: Shen Yinjie >Priority: Major > Attachments: YARN-6462_2.patch > > > we need a yarn command to list all leaf queues. > especially in large scale cluster ,there are a large amount of queues in > tree-format for various apps , we actually need a list-all-leaf-queues > interface to get queues infomation immediately ,other than search in fair > scheduler.xml or in yarn-scheduler web layer by layer.sometimes we should > also vertify a new queue is successfully added in scheduer, instead of fail > for some format error or other wise. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7827) Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
[
https://issues.apache.org/jira/browse/YARN-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357954#comment-16357954
]
Hudson commented on YARN-7827:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13637 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13637/])
YARN-7827. Stop and Delete Yarn Service from RM UI fails with HTTP ERROR
(jianhe: rev ddec08d7ccc8e43492fca2784203bd8af5e968cc)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-app/info.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/components/deploy-service.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/components/deploy-service.hbs
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-app/info.hbs
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-servicedef.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-deploy-service.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-app.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-app.hbs
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/utils/info-seeder.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/config/default-config.js
> Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
> -
>
> Key: YARN-7827
> URL: https://issues.apache.org/jira/browse/YARN-7827
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Yesha Vora
>Assignee: Sunil G
>Priority: Critical
> Attachments: Screen Shot 2018-02-07 at 3.51.54 PM.png,
> YARN-7827.001.patch, YARN-7827.002.patch
>
>
> Steps:
> 1) Enable Ats v2
> 2) Start Httpd Yarn service
> 3) Go to UI2 attempts page for yarn service
> 4) Click on setting icon
> 5) Click on stop service
> 6) This action will pop up a box to confirm stop. click on "Yes"
> Expected behavior:
> Yarn service should be stopped
> Actual behavior:
> Yarn UI is not notifying on whether Yarn service is stopped or not.
> On checking network stack trace, the PUT request failed with HTTP error 404
> {code}
> Sorry, got error 404
> Please consult RFC 2616 for meanings of the error code.
> Error Details
> org.apache.hadoop.yarn.webapp.WebAppException: /v1/services/httpd-hrt-qa-n:
> controller for v1 not found
> at org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:247)
> at org.apache.hadoop.yarn.webapp.Router.resolve(Router.java:155)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:143)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
> at
> com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110)
> at
>
[jira] [Commented] (YARN-7827) Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
[
https://issues.apache.org/jira/browse/YARN-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357948#comment-16357948
]
Sunil G commented on YARN-7827:
---
YARN-7912 is raised to handle KNOX scenario.
> Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
> -
>
> Key: YARN-7827
> URL: https://issues.apache.org/jira/browse/YARN-7827
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Yesha Vora
>Assignee: Sunil G
>Priority: Critical
> Attachments: Screen Shot 2018-02-07 at 3.51.54 PM.png,
> YARN-7827.001.patch, YARN-7827.002.patch
>
>
> Steps:
> 1) Enable Ats v2
> 2) Start Httpd Yarn service
> 3) Go to UI2 attempts page for yarn service
> 4) Click on setting icon
> 5) Click on stop service
> 6) This action will pop up a box to confirm stop. click on "Yes"
> Expected behavior:
> Yarn service should be stopped
> Actual behavior:
> Yarn UI is not notifying on whether Yarn service is stopped or not.
> On checking network stack trace, the PUT request failed with HTTP error 404
> {code}
> Sorry, got error 404
> Please consult RFC 2616 for meanings of the error code.
> Error Details
> org.apache.hadoop.yarn.webapp.WebAppException: /v1/services/httpd-hrt-qa-n:
> controller for v1 not found
> at org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:247)
> at org.apache.hadoop.yarn.webapp.Router.resolve(Router.java:155)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:143)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
> at
> com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1578)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>
[jira] [Commented] (YARN-7827) Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
[
https://issues.apache.org/jira/browse/YARN-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357941#comment-16357941
]
Jian He commented on YARN-7827:
---
I committed the patch to trunk, we can open separate jira to unblock the issue.
> Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
> -
>
> Key: YARN-7827
> URL: https://issues.apache.org/jira/browse/YARN-7827
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Yesha Vora
>Assignee: Sunil G
>Priority: Critical
> Attachments: Screen Shot 2018-02-07 at 3.51.54 PM.png,
> YARN-7827.001.patch, YARN-7827.002.patch
>
>
> Steps:
> 1) Enable Ats v2
> 2) Start Httpd Yarn service
> 3) Go to UI2 attempts page for yarn service
> 4) Click on setting icon
> 5) Click on stop service
> 6) This action will pop up a box to confirm stop. click on "Yes"
> Expected behavior:
> Yarn service should be stopped
> Actual behavior:
> Yarn UI is not notifying on whether Yarn service is stopped or not.
> On checking network stack trace, the PUT request failed with HTTP error 404
> {code}
> Sorry, got error 404
> Please consult RFC 2616 for meanings of the error code.
> Error Details
> org.apache.hadoop.yarn.webapp.WebAppException: /v1/services/httpd-hrt-qa-n:
> controller for v1 not found
> at org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:247)
> at org.apache.hadoop.yarn.webapp.Router.resolve(Router.java:155)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:143)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
> at
> com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1578)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
>
[jira] [Commented] (YARN-7655) Avoid AM preemption caused by RRs for specific nodes or racks
[
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357942#comment-16357942
]
Yufei Gu commented on YARN-7655:
aha, I found the same thing about getRelaxLocality, and just want to do the
same thing. Thanks for filing these two.
> Avoid AM preemption caused by RRs for specific nodes or racks
> -
>
> Key: YARN-7655
> URL: https://issues.apache.org/jira/browse/YARN-7655
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7655-001.patch, YARN-7655-002.patch,
> YARN-7655-003.patch, YARN-7655-004.patch
>
>
> We frequently see AM preemptions when
> {{starvedApp.getStarvedResourceRequests()}} in
> {{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs
> that request containers on a specific node. Since this causes us to only
> consider one node to preempt containers on, the really good work that was
> done in YARN-5830 doesn't save us from AM preemption. Even though there might
> be multiple nodes on which we could preempt enough non-AM containers to
> satisfy the app's starvation, we often wind up preempting one or more AM
> containers on the single node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM
> containers for an RR that specifies a node or rack, then we should instead
> expand the search space to consider all nodes. That way we take advantage of
> YARN-5830, and only preempt AMs if there's no alternative. I've attached a
> patch with an initial implementation of this. We've been running it on a few
> clusters, and have seen AM preemptions drop from double-digit occurrences on
> many days to zero.
> Of course, the tradeoff is some loss of locality, since the starved app is
> less likely to be allocated resources at the most specific locality level
> that it asked for. My opinion is that this tradeoff is worth it, but
> interested to hear what others think as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7912) While launching Native Service app from UI, consider service owner name from user.name query parameter
Sunil G created YARN-7912: - Summary: While launching Native Service app from UI, consider service owner name from user.name query parameter Key: YARN-7912 URL: https://issues.apache.org/jira/browse/YARN-7912 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Reporter: Sunil G As per comments from [~eyang] in YARN-7827, "For supporting knox, it would be good for javascript to detect the url entering /ui2 and process [user.name|http://user.name/] property. If there isn't one found, then proceed with ajax call to resource manager to find out who is the current user to pass the parameter along the rest api calls." This Jira will track to handle this. This is now pending feasibility check. Thanks [~eyang] and [~jianhe] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7835) [Atsv2] Race condition in NM while publishing events if second attempt launched on same node
[ https://issues.apache.org/jira/browse/YARN-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357939#comment-16357939 ] Rohith Sharma K S commented on YARN-7835: - bq. so I guess the code is meant to be thread safe. Since these methods are event driven, ideally this shouldn't be thread safe. But in our code i.e stopContainer schedules a thread that run after some time to remove application. This made me add synchronous block only for masterContainers set. I will change as per you comment which is thread safe forever. bq. Am I missing something? Since stopContainer schedules a thread default is 1sec where it removes the application from collectors. Without current fix it check for application in loop and come out once application is removed from collector. It fails in next assert because we are expecting this application to present. I guess waiting for 2 sec in loop should be fine, otherwise we can reduce it to 1.5 seconds. bq. I think we could reuse GenericTestUtils.waitFor(). It appears whole class to be refactored with this change. Let me see for feasibility. > [Atsv2] Race condition in NM while publishing events if second attempt > launched on same node > > > Key: YARN-7835 > URL: https://issues.apache.org/jira/browse/YARN-7835 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Critical > Attachments: YARN-7835.001.patch > > > It is observed race condition that if master container is killed for some > reason and launched on same node then NMTimelinePublisher doesn't add > timelineClient. But once completed container for 1st attempt has come then > NMTimelinePublisher removes the timelineClient. > It causes all subsequent event publishing from different client fails to > publish with exception Application is not found. ! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7827) Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
[
https://issues.apache.org/jira/browse/YARN-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357926#comment-16357926
]
Sunil G commented on YARN-7827:
---
Thanks [~eyang]
bq.It can be refined to an ajax call to resource manager to figure out who is
the currently connected user for simple security.
UI when it send the requests, server wont be knowing connected user. So as a
stop-gap solution till get a better model, I thought of having this option.
Does it make sense?
bq.For supporting knox, it would be good for javascript to detect the url
entering /ui2 and process user.name property.
This makes sense. I ll check that feasibility, but if its need more change, i
ll handle in other Jira so that this could be unblocked.
> Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
> -
>
> Key: YARN-7827
> URL: https://issues.apache.org/jira/browse/YARN-7827
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Yesha Vora
>Assignee: Sunil G
>Priority: Critical
> Attachments: Screen Shot 2018-02-07 at 3.51.54 PM.png,
> YARN-7827.001.patch, YARN-7827.002.patch
>
>
> Steps:
> 1) Enable Ats v2
> 2) Start Httpd Yarn service
> 3) Go to UI2 attempts page for yarn service
> 4) Click on setting icon
> 5) Click on stop service
> 6) This action will pop up a box to confirm stop. click on "Yes"
> Expected behavior:
> Yarn service should be stopped
> Actual behavior:
> Yarn UI is not notifying on whether Yarn service is stopped or not.
> On checking network stack trace, the PUT request failed with HTTP error 404
> {code}
> Sorry, got error 404
> Please consult RFC 2616 for meanings of the error code.
> Error Details
> org.apache.hadoop.yarn.webapp.WebAppException: /v1/services/httpd-hrt-qa-n:
> controller for v1 not found
> at org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:247)
> at org.apache.hadoop.yarn.webapp.Router.resolve(Router.java:155)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:143)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
> at
> com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1578)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
>
[jira] [Commented] (YARN-7903) Method getStarvedResourceRequests() only consider the first encountered resource
[
https://issues.apache.org/jira/browse/YARN-7903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357923#comment-16357923
]
Steven Rand commented on YARN-7903:
---
Agreed that having a concept of delay scheduling for preemption is a good idea
and would help with both JIRAs. We might be able to use
{{FSAppAttempt.getAllowedLocalityLevel}} or
{{FSAppAttempt.getAllowedLocalityLevelByTime}}, since those already have logic
for checking whether the app has waited longer than the threshold for requests
with some {{SchedulerKey}} (which seems to really just mean priority?). I'll
defer to others though on whether it makes sense for delay logic in preemption
to match delay logic in allocation -- possibly there are differences between
the two that call for separate logic.
I'm also quite confused as to how we should be thinking about different RRs
from the same app at the same priority. I spent some time digging through the
code today, but don't really understand it yet. There are a couple pieces of
code I found that deal with deduping/deconflicting RRs, but I wasn't sure how
to interpret them:
* {{VisitedResourceRequestTracker}} seems to consider RRs with the same
priority and capability to be logically the same
* {{AppSchedulingInfo#internalAddResourceRequests}} seems to consider RRs with
the same {{SchedulerRequestKey}} and resourceName to be logically the same
> Method getStarvedResourceRequests() only consider the first encountered
> resource
>
>
> Key: YARN-7903
> URL: https://issues.apache.org/jira/browse/YARN-7903
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Priority: Major
>
> We need to specify rack and ANY while submitting a node local resource
> request, as YARN-7561 discussed. For example:
> {code}
> ResourceRequest nodeRequest =
> createResourceRequest(GB, node1.getHostName(), 1, 1, false);
> ResourceRequest rackRequest =
> createResourceRequest(GB, node1.getRackName(), 1, 1, false);
> ResourceRequest anyRequest =
> createResourceRequest(GB, ResourceRequest.ANY, 1, 1, false);
> List resourceRequests =
> Arrays.asList(nodeRequest, rackRequest, anyRequest);
> {code}
> However, method getStarvedResourceRequests() only consider the first
> encountered resource, which most likely is ResourceRequest.ANY. That's a
> mismatch for locality request.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7655) Avoid AM preemption caused by RRs for specific nodes or racks
[
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357913#comment-16357913
]
Steven Rand commented on YARN-7655:
---
Thanks [~yufeigu]. I filed YARN-7910 for the {{TODO}} in the unit test.
Unfortunately I also realized that I made a mistake in how I interpreted the
value of {{ResourceRequest.getRelaxLocality}} -- filed YARN-7911 for that.
> Avoid AM preemption caused by RRs for specific nodes or racks
> -
>
> Key: YARN-7655
> URL: https://issues.apache.org/jira/browse/YARN-7655
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7655-001.patch, YARN-7655-002.patch,
> YARN-7655-003.patch, YARN-7655-004.patch
>
>
> We frequently see AM preemptions when
> {{starvedApp.getStarvedResourceRequests()}} in
> {{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs
> that request containers on a specific node. Since this causes us to only
> consider one node to preempt containers on, the really good work that was
> done in YARN-5830 doesn't save us from AM preemption. Even though there might
> be multiple nodes on which we could preempt enough non-AM containers to
> satisfy the app's starvation, we often wind up preempting one or more AM
> containers on the single node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM
> containers for an RR that specifies a node or rack, then we should instead
> expand the search space to consider all nodes. That way we take advantage of
> YARN-5830, and only preempt AMs if there's no alternative. I've attached a
> patch with an initial implementation of this. We've been running it on a few
> clusters, and have seen AM preemptions drop from double-digit occurrences on
> many days to zero.
> Of course, the tradeoff is some loss of locality, since the starved app is
> less likely to be allocated resources at the most specific locality level
> that it asked for. My opinion is that this tradeoff is worth it, but
> interested to hear what others think as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7911) Method identifyContainersToPreempt uses ResourceRequest#getRelaxLocality incorrectly
Steven Rand created YARN-7911:
-
Summary: Method identifyContainersToPreempt uses
ResourceRequest#getRelaxLocality incorrectly
Key: YARN-7911
URL: https://issues.apache.org/jira/browse/YARN-7911
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler, resourcemanager
Affects Versions: 3.1.0
Reporter: Steven Rand
Assignee: Steven Rand
After YARN-7655, in {{identifyContainersToPreempt}} we expand the search space
to all nodes if we had previously only considered a subset to satisfy a
{{NODE_LOCAL}} or {{RACK_LOCAL}} RR, and were going to preempt AM containers as
a result, and the RR allowed locality to be relaxed:
{code}
// Don't preempt AM containers just to satisfy local requests if relax
// locality is enabled.
if (bestContainers != null
&& bestContainers.numAMContainers > 0
&& !ResourceRequest.isAnyLocation(rr.getResourceName())
&& rr.getRelaxLocality()) {
bestContainers = identifyContainersToPreemptForOneContainer(
scheduler.getNodeTracker().getAllNodes(), rr);
}
{code}
This turns out to be based on a misunderstanding of what
{{rr.getRelaxLocality}} means. I had believed that it means that locality can
be relaxed _from_ that level. However, it actually means that locality can be
relaxed _to_ that level:
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java#L450.
For example, suppose we have {{relaxLocality}} set to {{true}} at the node
level, but {{false}} at the rack and {{ANY}} levels. This is saying that we
cannot relax locality to the rack level. However, the current behavior after
YARN-7655 is to interpret relaxLocality being true at the node level as saying
that it's okay to satisfy the request elsewhere.
What we should do instead is check whether relaxLocality is enabled for the
corresponding RR at the next level. So if we're considering a node-level RR, we
should find the corresponding rack-level RR and check whether relaxLocality is
enabled for it. And similarly, if we're considering a rack-level RR, we should
check the corresponding any-level RR.
It may also be better to use {{FSAppAttempt#getAllowedLocalityLevel}} instead
of explicitly checking {{relaxLocality}}, but I'm not sure which is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7910) Fix TODO in TestFairSchedulerPreemption#testRelaxLocalityToNotPreemptAM
Steven Rand created YARN-7910:
-
Summary: Fix TODO in
TestFairSchedulerPreemption#testRelaxLocalityToNotPreemptAM
Key: YARN-7910
URL: https://issues.apache.org/jira/browse/YARN-7910
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler, test
Affects Versions: 3.1.0
Reporter: Steven Rand
Assignee: Steven Rand
In YARN-7655, we left a {{TODO}} in the newly added test:
{code}
// TODO (YARN-7655) The starved app should be allocated 4 containers.
// It should be possible to modify the RRs such that this is true
// after YARN-7903.
verifyPreemption(0, 4);
{code}
This JIRA is to track resolving that after YARN-7903 is resolved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7892) NodeAttributePBImpl does not implement hashcode and Equals properly
[
https://issues.apache.org/jira/browse/YARN-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357876#comment-16357876
]
Weiwei Yang commented on YARN-7892:
---
Hi [~Naganarasimha], [~sunilg]
I think we should check both prefix/key/name/value in equals (hashcode),
otherwise how we can tell if two attributes are same? For example if we have
following attributes on h1 and h2,
{code:java}
h1: A=A1
h2: A=A2
{code}
In attribute-manager, we use node-attribute as a key that maps to a list of
nodes have this attribute,
attribute -> nodes
with this patch, the mapping will be like
{code:java}
A: [h1, h2]
{code}
but when we do placement decision, we want to put a container on only nodes
where {{A=A1}}, then how can this be done?
Regarding to [~Naganarasimha]'s concern
{quote}Else in every place we need to separately start storing with Name and
prefix what are the attributes for example 7856 is newly introducing a map to
just compare whether any duplicates exists.
{quote}
This can be resolved by an utility class method, e.g
{{NodeAttributeUtils#sameAttributeKey(NodeAttribute na1, NodeAttribute na2)}},
or something similar, not a big problem.
Does this make sense?
Thanks.
> NodeAttributePBImpl does not implement hashcode and Equals properly
> ---
>
> Key: YARN-7892
> URL: https://issues.apache.org/jira/browse/YARN-7892
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
> Attachments: YARN-7892-YARN-3409.001.patch,
> YARN-7892-YARN-3409.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7909) YARN service REST API returns charset=null when kerberos enabled
[
https://issues.apache.org/jira/browse/YARN-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357860#comment-16357860
]
genericqa commented on YARN-7909:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7909 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909869/YARN-7909.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 1b6e9ecb6f59 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1bc03dd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19648/testReport/ |
| Max. process+thread count | 441 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19648/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT
[jira] [Updated] (YARN-7909) YARN service REST API returns charset=null when kerberos enabled
[
https://issues.apache.org/jira/browse/YARN-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7909:
Attachment: YARN-7909.001.patch
> YARN service REST API returns charset=null when kerberos enabled
>
>
> Key: YARN-7909
> URL: https://issues.apache.org/jira/browse/YARN-7909
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-7909.001.patch
>
>
> When kerberos is enabled, the response header will reply with:
> {code:java}
> Content-Type: application/json;charset=null{code}
> This appears to be somehow AuthenticationFilter is trigger charset to become
> undefined.
> The same problem is not observed when simple security is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7906) mvn site fails
[
https://issues.apache.org/jira/browse/YARN-7906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357834#comment-16357834
]
Akira Ajisaka commented on YARN-7906:
-
As Sunil said, {{org.apache.hadoop.yarn.api.records.impl.pb}} package-info.java
has the same content, however, it is not a problem. The problem is that a
package is annotated more than once in package-info.java. There are two
package-info.java for {{org.apache.hadoop.yarn.client.api.impl}} package in
hadoop-yarn-common module and hadoop-yarn-api module and the both has
{{@Private}} annotation. The patch is to remove {{@Private}} from
hadoop-yarn-api package.
> mvn site fails
> --
>
> Key: YARN-7906
> URL: https://issues.apache.org/jira/browse/YARN-7906
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build, documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Blocker
> Attachments: YARN-7906.001.patch
>
>
> {{mvn site}} fails on trunk.
> {noformat}
> [ERROR] javadoc: warning - Multiple sources of package comments found for
> package "org.apache.hadoop.yarn.client.api.impl"
> [ERROR]
> /home/travis/build/aajisaka/hadoop-document/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/resource/package-info.java:21:
> error: package org.apache.hadoop.yarn.api.resource has already been annotated
> [ERROR] @InterfaceAudience.Private
> [ERROR] ^
> [ERROR] java.lang.AssertionError
> [ERROR] at com.sun.tools.javac.util.Assert.error(Assert.java:126)
> [ERROR] at com.sun.tools.javac.util.Assert.check(Assert.java:45)
> [ERROR] at
> com.sun.tools.javac.code.SymbolMetadata.setDeclarationAttributesWithCompletion(SymbolMetadata.java:161)
> [ERROR] at
> com.sun.tools.javac.code.Symbol.setDeclarationAttributesWithCompletion(Symbol.java:215)
> [ERROR] at
> com.sun.tools.javac.comp.MemberEnter.actualEnterAnnotations(MemberEnter.java:952)
> [ERROR] at
> com.sun.tools.javac.comp.MemberEnter.access$600(MemberEnter.java:64)
> [ERROR] at com.sun.tools.javac.comp.MemberEnter$5.run(MemberEnter.java:876)
> [ERROR] at com.sun.tools.javac.comp.Annotate.flush(Annotate.java:143)
> [ERROR] at com.sun.tools.javac.comp.Annotate.enterDone(Annotate.java:129)
> [ERROR] at com.sun.tools.javac.comp.Enter.complete(Enter.java:512)
> [ERROR] at com.sun.tools.javac.comp.Enter.main(Enter.java:471)
> [ERROR] at com.sun.tools.javadoc.JavadocEnter.main(JavadocEnter.java:78)
> [ERROR] at
> com.sun.tools.javadoc.JavadocTool.getRootDocImpl(JavadocTool.java:186)
> [ERROR] at com.sun.tools.javadoc.Start.parseAndExecute(Start.java:346)
> [ERROR] at com.sun.tools.javadoc.Start.begin(Start.java:219)
> [ERROR] at com.sun.tools.javadoc.Start.begin(Start.java:205)
> [ERROR] at com.sun.tools.javadoc.Main.execute(Main.java:64)
> [ERROR] at com.sun.tools.javadoc.Main.main(Main.java:54)
> [ERROR] javadoc: error - fatal error
> {noformat}
> [https://travis-ci.org/aajisaka/hadoop-document/builds/338833122|https://travis-ci.org/aajisaka/hadoop-document/builds/338833122]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-7854) Attach prefixes to different type of node attributes
[
https://issues.apache.org/jira/browse/YARN-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7854:
-
Comment: was deleted
(was: Thanks [~LambertYe] for investigations. Few general comments.
1. {{yarn_rm_admin/}} is not what we look for namespace. We wanted it to look
like more of like a domain / namespace. Hence {{system.yarn.io/}} kind of name
tag which is what [~leftnoteasy] also was indicating earlier. If you see the
new resource types, we are putting under {{yarn.io/gpu}} or {{yarn.io/fpga}}
etc. So could we have like {{admin.yarn.io}} {{dist.yarn.io}}
{{system.yarn.io}} ? please feel free to suggest ur thoughts.
2. While an app demands a constraint for attributes, they have to specify the
fully qualified name with prefix.
3. If prefix is not given, we can think of looking under centralized namespace
alone.
4. cross validation from RM makes sense as [~Naganarasimha] mentioned.)
> Attach prefixes to different type of node attributes
>
>
> Key: YARN-7854
> URL: https://issues.apache.org/jira/browse/YARN-7854
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: RM
>Reporter: Weiwei Yang
>Assignee: LiangYe
>Priority: Major
>
> There are multiple types of node attributes depending on which source it
> comes from, includes
> # Centralized: attributes set by users (admin or normal users)
> # Distributed: attributes collected by a certain attribute provider on each
> NM
> # System: some built-in attributes in yarn, set by yarn internal components,
> e.g scheduler
> To better manage these attributes, we introduce the prefix (namespace)
> concept to the an attribute. This Jira is opened to figure out how to attach
> prefixes (automatically/implicitly or explicitly) to different type of
> attributes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-7854) Attach prefixes to different type of node attributes
[
https://issues.apache.org/jira/browse/YARN-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7854:
-
Comment: was deleted
(was: In k8s，
[Namespace|https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/namespaces.md]
is a mechanism to partition resources created by users into logically named
group.
The
[Namespace|https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/]
provides a unique scope for:
# named resources (pods, services, replication controllers, etc. to avoid
basic naming collisions, {color:#ff}*now not include nodes and
persistantVolume*{color})
# delegated management authority to trusted users
# ability to limit community resource consumption
In k8s, _Labels_ are key/value pairs that are attached to objects, such as
pods, nodes. Valid label keys have two segments: an optional prefix and name,
separated by a slash ({{/}}). If the prefix is omitted, the label Key is
presumed to be private to the user. Automated system components (e.g.
{{kube-scheduler}}, {{kube-controller-manager}}, {{kube-apiserver}},
{{kubectl}}, or other third-party automation) which add labels to end-user
objects must specify a prefix. The {{kubernetes.io/}} prefix is reserved for
Kubernetes core components.
In k8s, according to different object's labels, there is different affinity,
e.g. [node labels——node
affinity|https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/nodeaffinity.md],
[pod labels——pod
affinity|https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/podaffinity.md].
Especially [node taints——pod
toleration|https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/taint-toleration-dedicated.md].
In yarn, according to YARN-3409 YARN-6592 [#YARN-7800] and so, node
attribute is like k8s node labels, allocation tags to container is like k8s pod
labels but in k8s pod labels not delivery to nodes for namespace isolation,
node constraint is some kind of k8s taints and tolerations.
So to this Jira issue, to make up prefix for node attributes from different
sources is a kind of namespace isolation.
For now,
# Centralized: attributes set by users (admin or normal users) can use a
prefix like {color:#ff}yarn_rm_admin/{color}
# Distributed: attributes collected by a certain attribute provider on each NM
can use a prefix like {color:#ff}yarn_nm/{color}
# System: some built-in attributes in yarn, set by yarn internal components,
e.g scheduler like {color:#ff}yarn_system/{color}
then,
In RM merge from these sources, no need to handle the conflict cases.
In placement constraints for node attributes should specify prefix to indicate
the sources/namespaces, or use yarn_nm/ prefix by default.
[~cheersyang] [~leftnoteasy] [~Naganarasimha] any comment? Thanks.)
> Attach prefixes to different type of node attributes
>
>
> Key: YARN-7854
> URL: https://issues.apache.org/jira/browse/YARN-7854
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: RM
>Reporter: Weiwei Yang
>Assignee: LiangYe
>Priority: Major
>
> There are multiple types of node attributes depending on which source it
> comes from, includes
> # Centralized: attributes set by users (admin or normal users)
> # Distributed: attributes collected by a certain attribute provider on each
> NM
> # System: some built-in attributes in yarn, set by yarn internal components,
> e.g scheduler
> To better manage these attributes, we introduce the prefix (namespace)
> concept to the an attribute. This Jira is opened to figure out how to attach
> prefixes (automatically/implicitly or explicitly) to different type of
> attributes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-7854) Attach prefixes to different type of node attributes
[ https://issues.apache.org/jira/browse/YARN-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7854: - Comment: was deleted (was: Thanks [~sunilg] [~cheersyang] for comments. # Agree, use DNS format is more appropriate. # Advocate to Specify the fully qualified name. # If prefix is not given, default namespace can be configured by configuration? Because different organizations have different requirements. # If cross validation is between different sources and they are in different namespaces, it's not make sense. If from the same source as [~cheersyang] mentioned, nm collectors or admin or system should handle this situation and make them the same type or definiteness.) > Attach prefixes to different type of node attributes > > > Key: YARN-7854 > URL: https://issues.apache.org/jira/browse/YARN-7854 > Project: Hadoop YARN > Issue Type: Sub-task > Components: RM >Reporter: Weiwei Yang >Assignee: LiangYe >Priority: Major > > There are multiple types of node attributes depending on which source it > comes from, includes > # Centralized: attributes set by users (admin or normal users) > # Distributed: attributes collected by a certain attribute provider on each > NM > # System: some built-in attributes in yarn, set by yarn internal components, > e.g scheduler > To better manage these attributes, we introduce the prefix (namespace) > concept to the an attribute. This Jira is opened to figure out how to attach > prefixes (automatically/implicitly or explicitly) to different type of > attributes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-7854) Attach prefixes to different type of node attributes
[ https://issues.apache.org/jira/browse/YARN-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7854: - Comment: was deleted (was: Hi [~LambertYe] Thanks for the investigation. Following was from an earlier discussion bq. Prefix will help in segregating the attributes provided by different sources and is a DNS name, like nm.yarn.io/docker-supported,nm.yarn.io/container-executor-type, *.yarn.io is limited to be defined by system instead of admin using centralized API. So we need to set prefix name with DNS format. bq. In RM merge from these sources, no need to handle the conflict cases. This might be still necessary. [~Naganarasimha] raised his concern this morning about, for example, nm1 reports an attribute A:STRING:V1, nm2 reports an attribute A:FLOAT:V2, is this allowed? He suggested we need some cross-validation in RM on types. [~Naganarasimha], could you please comment your idea? bq. In placement constraints for node attributes should specify prefix to indicate the sources/namespaces, or use yarn_nm/ prefix by default. I agree. We need to define such behaviors. A rough thought is that in first phase, we can ignore namespace check, that means an user is able to see all attributes (this is related to constraints so we can visit this later). User needs to specify the prefix while using it (so it is able to get a match), otherwise a default prefix is added which belongs to this user's namespace (I don't think we should use yarn_nm as default prefix for this case). Thanks) > Attach prefixes to different type of node attributes > > > Key: YARN-7854 > URL: https://issues.apache.org/jira/browse/YARN-7854 > Project: Hadoop YARN > Issue Type: Sub-task > Components: RM >Reporter: Weiwei Yang >Assignee: LiangYe >Priority: Major > > There are multiple types of node attributes depending on which source it > comes from, includes > # Centralized: attributes set by users (admin or normal users) > # Distributed: attributes collected by a certain attribute provider on each > NM > # System: some built-in attributes in yarn, set by yarn internal components, > e.g scheduler > To better manage these attributes, we introduce the prefix (namespace) > concept to the an attribute. This Jira is opened to figure out how to attach > prefixes (automatically/implicitly or explicitly) to different type of > attributes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7854) Attach prefixes to different type of node attributes
[ https://issues.apache.org/jira/browse/YARN-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan resolved YARN-7854. -- Resolution: Later > Attach prefixes to different type of node attributes > > > Key: YARN-7854 > URL: https://issues.apache.org/jira/browse/YARN-7854 > Project: Hadoop YARN > Issue Type: Sub-task > Components: RM >Reporter: Weiwei Yang >Assignee: LiangYe >Priority: Major > > There are multiple types of node attributes depending on which source it > comes from, includes > # Centralized: attributes set by users (admin or normal users) > # Distributed: attributes collected by a certain attribute provider on each > NM > # System: some built-in attributes in yarn, set by yarn internal components, > e.g scheduler > To better manage these attributes, we introduce the prefix (namespace) > concept to the an attribute. This Jira is opened to figure out how to attach > prefixes (automatically/implicitly or explicitly) to different type of > attributes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7909) YARN service REST API returns charset=null when kerberos enabled
Eric Yang created YARN-7909:
---
Summary: YARN service REST API returns charset=null when kerberos
enabled
Key: YARN-7909
URL: https://issues.apache.org/jira/browse/YARN-7909
Project: Hadoop YARN
Issue Type: Bug
Components: yarn-native-services
Affects Versions: 3.1.0
Reporter: Eric Yang
Assignee: Eric Yang
When kerberos is enabled, the response header will reply with:
{code:java}
Content-Type: application/json;charset=null{code}
This appears to be somehow AuthenticationFilter is trigger charset to become
undefined.
The same problem is not observed when simple security is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7881) Add Log Aggregation Status API to the RM Webservice
[
https://issues.apache.org/jira/browse/YARN-7881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357728#comment-16357728
]
Giovanni Matteo Fumarola commented on YARN-7881:
Thanks [~GergelyNovak] for the patch. Everything looks good from my review.
Can you add some tests for {{RMWebServices}} changes?
> Add Log Aggregation Status API to the RM Webservice
> ---
>
> Key: YARN-7881
> URL: https://issues.apache.org/jira/browse/YARN-7881
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: yarn
>Reporter: Gergely Novák
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7881.001.patch, YARN-7881.002.patch
>
>
> The old YARN UI has a page: /cluster/logaggregationstatus/\{app_id} which
> shows the log aggregation status for all the nodes that run containers for
> the given application. In order to add a similar page to the new YARN UI we
> need to add an RM WS endpoint first.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[
https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated YARN-7451:
Target Version/s: 3.0.2 (was: 3.0.1)
> Resources Types should be visible in the Cluster Apps API "resourceRequests"
> section
>
>
> Key: YARN-7451
> URL: https://issues.apache.org/jira/browse/YARN-7451
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, restapi
>Affects Versions: 3.0.0
>Reporter: Grant Sohn
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-7451.001.patch, YARN-7451.002.patch,
> YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch,
> YARN-7451.006.patch, YARN-7451.007.patch, YARN-7451.008.patch,
> YARN-7451.009.patch, YARN-7451.010.patch,
> YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch
>
>
> When running jobs that request resource types the RM Cluster Apps API should
> include this in the "resourceRequests" object.
> Additionally, when calling the RM scheduler API it returns:
> {noformat}
> "childQueues": {
> "queue": [
> {
> "allocatedContainers": 101,
> "amMaxResources": {
> "memory": 320390,
> "vCores": 192
> },
> "amUsedResources": {
> "memory": 1024,
> "vCores": 1
> },
> "clusterResources": {
> "memory": 640779,
> "vCores": 384
> },
> "demandResources": {
> "memory": 103424,
> "vCores": 101
> },
> "fairResources": {
> "memory": 640779,
> "vCores": 384
> },
> "maxApps": 2147483647,
> "maxResources": {
> "memory": 640779,
> "vCores": 384
> },
> "minResources": {
> "memory": 0,
> "vCores": 0
> },
> "numActiveApps": 1,
> "numPendingApps": 0,
> "preemptable": true,
> "queueName": "root.users.systest",
> "reservedContainers": 0,
> "reservedResources": {
> "memory": 0,
> "vCores": 0
> },
> "schedulingPolicy": "fair",
> "steadyFairResources": {
> "memory": 320390,
> "vCores": 192
> },
> "type": "fairSchedulerLeafQueueInfo",
> "usedResources": {
> "memory": 103424,
> "vCores": 101
> }
> }
> ]
> {noformat}
> However, the web UI shows resource types usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7649) RMContainer state transition exception after container update
[
https://issues.apache.org/jira/browse/YARN-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated YARN-7649:
Target Version/s: 2.9.1, 3.0.2 (was: 2.9.1, 3.0.1)
> RMContainer state transition exception after container update
> -
>
> Key: YARN-7649
> URL: https://issues.apache.org/jira/browse/YARN-7649
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Weiwei Yang
>Assignee: Arun Suresh
>Priority: Major
>
> I've been seen this in a cluster deployment as well as in UT, run
> {{TestAMRMClient#testAMRMClientWithContainerPromotion}} could reproduce this,
> it doesn't fail the test case but following error message is shown up in the
> log
> {noformat}
> 2017-12-13 19:41:31,817 ERROR rmcontainer.RMContainerImpl
> (RMContainerImpl.java:handle(480)) - Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> RELEASED at ALLOCATED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:478)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:65)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.completedContainer(AbstractYarnScheduler.java:675)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1586)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:155)
> at
> org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
> at java.lang.Thread.run(Thread.java:748)
> 2017-12-13 19:41:31,817 ERROR rmcontainer.RMContainerImpl
> (RMContainerImpl.java:handle(481)) - Invalid event RELEASED on container
> container_1513165290804_0001_01_03
> {noformat}
> this seems to be related to YARN-6251.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6956) preemption may only consider resource requests for one node
[
https://issues.apache.org/jira/browse/YARN-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357675#comment-16357675
]
genericqa commented on YARN-6956:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
56s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 9 new + 75 unchanged - 0 fixed = 84 total (was 75) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 50s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}132m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-6956 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881664/YARN-6956.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 4cab04a46d7e 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1bc03dd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19647/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Commented] (YARN-7827) Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
[
https://issues.apache.org/jira/browse/YARN-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357619#comment-16357619
]
Eric Yang commented on YARN-7827:
-
[~sunilg] For supporting knox, it would be good for javascript to detect the
url entering /ui2 and process user.name property. If there isn't one found,
then proceed with ajax call to resource manager to find out who is the current
user to pass the parameter along the rest api calls.
> Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
> -
>
> Key: YARN-7827
> URL: https://issues.apache.org/jira/browse/YARN-7827
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Yesha Vora
>Assignee: Sunil G
>Priority: Critical
> Attachments: Screen Shot 2018-02-07 at 3.51.54 PM.png,
> YARN-7827.001.patch, YARN-7827.002.patch
>
>
> Steps:
> 1) Enable Ats v2
> 2) Start Httpd Yarn service
> 3) Go to UI2 attempts page for yarn service
> 4) Click on setting icon
> 5) Click on stop service
> 6) This action will pop up a box to confirm stop. click on "Yes"
> Expected behavior:
> Yarn service should be stopped
> Actual behavior:
> Yarn UI is not notifying on whether Yarn service is stopped or not.
> On checking network stack trace, the PUT request failed with HTTP error 404
> {code}
> Sorry, got error 404
> Please consult RFC 2616 for meanings of the error code.
> Error Details
> org.apache.hadoop.yarn.webapp.WebAppException: /v1/services/httpd-hrt-qa-n:
> controller for v1 not found
> at org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:247)
> at org.apache.hadoop.yarn.webapp.Router.resolve(Router.java:155)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:143)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
> at
> com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1578)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
>
[jira] [Commented] (YARN-7813) Capacity Scheduler Intra-queue Preemption should be configurable for each queue
[
https://issues.apache.org/jira/browse/YARN-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357610#comment-16357610
]
genericqa commented on YARN-7813:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
13s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 7s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 9 new + 871 unchanged - 1 fixed = 880 total (was 872) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
11s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 6s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 28s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
|
[jira] [Commented] (YARN-7827) Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
[
https://issues.apache.org/jira/browse/YARN-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357608#comment-16357608
]
Eric Yang commented on YARN-7827:
-
Hi [~sunilg],
Thank you for the patch. Overall, the patch is in the right direction. One
nit pick about having User Name for service, this properly should not be a user
input. It can be refined to an ajax call to resource manager to figure out who
is the currently connected user for simple security.
> Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
> -
>
> Key: YARN-7827
> URL: https://issues.apache.org/jira/browse/YARN-7827
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Yesha Vora
>Assignee: Sunil G
>Priority: Critical
> Attachments: Screen Shot 2018-02-07 at 3.51.54 PM.png,
> YARN-7827.001.patch, YARN-7827.002.patch
>
>
> Steps:
> 1) Enable Ats v2
> 2) Start Httpd Yarn service
> 3) Go to UI2 attempts page for yarn service
> 4) Click on setting icon
> 5) Click on stop service
> 6) This action will pop up a box to confirm stop. click on "Yes"
> Expected behavior:
> Yarn service should be stopped
> Actual behavior:
> Yarn UI is not notifying on whether Yarn service is stopped or not.
> On checking network stack trace, the PUT request failed with HTTP error 404
> {code}
> Sorry, got error 404
> Please consult RFC 2616 for meanings of the error code.
> Error Details
> org.apache.hadoop.yarn.webapp.WebAppException: /v1/services/httpd-hrt-qa-n:
> controller for v1 not found
> at org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:247)
> at org.apache.hadoop.yarn.webapp.Router.resolve(Router.java:155)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:143)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
> at
> com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1578)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
>
[jira] [Resolved] (YARN-7907) Yarn app CLI client does not send Kerberos header to Resource Manager rest API
[
https://issues.apache.org/jira/browse/YARN-7907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved YARN-7907.
-
Resolution: Not A Bug
Assignee: Eric Yang
> Yarn app CLI client does not send Kerberos header to Resource Manager rest API
> --
>
> Key: YARN-7907
> URL: https://issues.apache.org/jira/browse/YARN-7907
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Eric Yang
>Priority: Critical
>
> Launch of yarn service app is failing in secure mode with below stacktrace.
> {code:java}
> [hrt_qa@xxx root]$ kinit -kt
> /home/hrt_qa/hadoopqa/keytabs/hrt_qa.headless.keytab hrt_qa
> [hrt_qa@xxx root]$ yarn app -launch test2 sleeper
> WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of
> YARN_LOG_DIR.
> WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of
> YARN_LOGFILE.
> WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of
> YARN_PID_DIR.
> WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
> 18/02/07 22:50:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/02/07 22:50:41 INFO client.AHSProxy: Connecting to Application History
> server at xxx/xxx:10200
> 18/02/07 22:50:41 INFO client.AHSProxy: Connecting to Application History
> server at xxx/xxx:10200
> 18/02/07 22:50:41 INFO client.ApiServiceClient: Loading service definition
> from local FS:
> /usr/hdp/3.0.0.0-800/hadoop-yarn/yarn-service-examples/sleeper/sleeper.json
> 18/02/07 22:50:42 ERROR client.ApiServiceClient: Authentication required{code}
> CLI client does not send Kerberos header to Resource Manager rest API.
> Tcpdump indicate that there is no token being sent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7907) Yarn app CLI client does not send Kerberos header to Resource Manager rest API
[
https://issues.apache.org/jira/browse/YARN-7907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357587#comment-16357587
]
Eric Yang commented on YARN-7907:
-
This problem can happen due to the following reasons:
# If user is not authorized in hadoop-policy.xml, it will prevent SPNEGO token
to be sent from client to server side.
# Hard code zookeeper.sasl.client.username=zookeeper in YARN_OPTS in
yarn-env.sh. User is unable to read zookeeper znodes to find active resource
manager.
Both problems are configuration mistakes, and they are not bugs in the current
YARN code. Therefore, close this as working as designed.
> Yarn app CLI client does not send Kerberos header to Resource Manager rest API
> --
>
> Key: YARN-7907
> URL: https://issues.apache.org/jira/browse/YARN-7907
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Priority: Critical
>
> Launch of yarn service app is failing in secure mode with below stacktrace.
> {code:java}
> [hrt_qa@xxx root]$ kinit -kt
> /home/hrt_qa/hadoopqa/keytabs/hrt_qa.headless.keytab hrt_qa
> [hrt_qa@xxx root]$ yarn app -launch test2 sleeper
> WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of
> YARN_LOG_DIR.
> WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of
> YARN_LOGFILE.
> WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of
> YARN_PID_DIR.
> WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
> 18/02/07 22:50:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/02/07 22:50:41 INFO client.AHSProxy: Connecting to Application History
> server at xxx/xxx:10200
> 18/02/07 22:50:41 INFO client.AHSProxy: Connecting to Application History
> server at xxx/xxx:10200
> 18/02/07 22:50:41 INFO client.ApiServiceClient: Loading service definition
> from local FS:
> /usr/hdp/3.0.0.0-800/hadoop-yarn/yarn-service-examples/sleeper/sleeper.json
> 18/02/07 22:50:42 ERROR client.ApiServiceClient: Authentication required{code}
> CLI client does not send Kerberos header to Resource Manager rest API.
> Tcpdump indicate that there is no token being sent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7655) Avoid AM preemption caused by RRs for specific nodes or racks
[
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357553#comment-16357553
]
Hudson commented on YARN-7655:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13636 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13636/])
YARN-7655. Avoid AM preemption caused by RRs for specific nodes or (yufei: rev
1bc03ddf97f3f0e0ecc1b00217438d3c91d29be5)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSPreemptionThread.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
> Avoid AM preemption caused by RRs for specific nodes or racks
> -
>
> Key: YARN-7655
> URL: https://issues.apache.org/jira/browse/YARN-7655
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7655-001.patch, YARN-7655-002.patch,
> YARN-7655-003.patch, YARN-7655-004.patch
>
>
> We frequently see AM preemptions when
> {{starvedApp.getStarvedResourceRequests()}} in
> {{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs
> that request containers on a specific node. Since this causes us to only
> consider one node to preempt containers on, the really good work that was
> done in YARN-5830 doesn't save us from AM preemption. Even though there might
> be multiple nodes on which we could preempt enough non-AM containers to
> satisfy the app's starvation, we often wind up preempting one or more AM
> containers on the single node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM
> containers for an RR that specifies a node or rack, then we should instead
> expand the search space to consider all nodes. That way we take advantage of
> YARN-5830, and only preempt AMs if there's no alternative. I've attached a
> patch with an initial implementation of this. We've been running it on a few
> clusters, and have seen AM preemptions drop from double-digit occurrences on
> many days to zero.
> Of course, the tradeoff is some loss of locality, since the starved app is
> less likely to be allocated resources at the most specific locality level
> that it asked for. My opinion is that this tradeoff is worth it, but
> interested to hear what others think as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7265) Hadoop Server Log Correlation
[ https://issues.apache.org/jira/browse/YARN-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357531#comment-16357531 ] Arun C Murthy commented on YARN-7265: - [~tanping] - Like [~jlowe] suggested, separating this from YARN makes sense. For e.g. Ambari has "log search", leveraging that would be a better option. Makes sense? Thanks. > Hadoop Server Log Correlation > --- > > Key: YARN-7265 > URL: https://issues.apache.org/jira/browse/YARN-7265 > Project: Hadoop YARN > Issue Type: Wish > Components: log-aggregation >Reporter: Tanping Wang >Priority: Major > > Hadoop has many server logs, yarn tasks logs, node manager logs, HDFS logs.. > There are also a lot of different ways can be used to expose the logs, build > relationship horizontally to correlate the logs or search the logs by > keyword. There is a need for a default and yet convenient logging analytics > mechanism in Hadoop itself that at least covers all the server logs of > Hadoop. This log analytics system can correlate the Hadoop server logs by > grouping them by various dimensions including application ID, task ID, job > ID or node ID etc. The raw logs with correlation can be easily accessed by > the application developer or cluster administrator via web page for managing > and debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7655) Avoid AM preemption caused by RRs for specific nodes or racks
[
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357528#comment-16357528
]
Yufei Gu commented on YARN-7655:
+1 for the last patch and committed to trunk. Thanks for the patch, [~Steven
Rand]. We can move the discussion to YARN-7903. Better to file an Jira to fix
the TODO in the unit test which is blocked by YARN-7903.
> Avoid AM preemption caused by RRs for specific nodes or racks
> -
>
> Key: YARN-7655
> URL: https://issues.apache.org/jira/browse/YARN-7655
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7655-001.patch, YARN-7655-002.patch,
> YARN-7655-003.patch, YARN-7655-004.patch
>
>
> We frequently see AM preemptions when
> {{starvedApp.getStarvedResourceRequests()}} in
> {{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs
> that request containers on a specific node. Since this causes us to only
> consider one node to preempt containers on, the really good work that was
> done in YARN-5830 doesn't save us from AM preemption. Even though there might
> be multiple nodes on which we could preempt enough non-AM containers to
> satisfy the app's starvation, we often wind up preempting one or more AM
> containers on the single node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM
> containers for an RR that specifies a node or rack, then we should instead
> expand the search space to consider all nodes. That way we take advantage of
> YARN-5830, and only preempt AMs if there's no alternative. I've attached a
> patch with an initial implementation of this. We've been running it on a few
> clusters, and have seen AM preemptions drop from double-digit occurrences on
> many days to zero.
> Of course, the tradeoff is some loss of locality, since the starved app is
> less likely to be allocated resources at the most specific locality level
> that it asked for. My opinion is that this tradeoff is worth it, but
> interested to hear what others think as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7655) Avoid AM preemption caused by RRs for specific nodes or racks
[
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-7655:
---
Summary: Avoid AM preemption caused by RRs for specific nodes or racks
(was: avoid AM preemption caused by RRs for specific nodes or racks)
> Avoid AM preemption caused by RRs for specific nodes or racks
> -
>
> Key: YARN-7655
> URL: https://issues.apache.org/jira/browse/YARN-7655
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Attachments: YARN-7655-001.patch, YARN-7655-002.patch,
> YARN-7655-003.patch, YARN-7655-004.patch
>
>
> We frequently see AM preemptions when
> {{starvedApp.getStarvedResourceRequests()}} in
> {{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs
> that request containers on a specific node. Since this causes us to only
> consider one node to preempt containers on, the really good work that was
> done in YARN-5830 doesn't save us from AM preemption. Even though there might
> be multiple nodes on which we could preempt enough non-AM containers to
> satisfy the app's starvation, we often wind up preempting one or more AM
> containers on the single node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM
> containers for an RR that specifies a node or rack, then we should instead
> expand the search space to consider all nodes. That way we take advantage of
> YARN-5830, and only preempt AMs if there's no alternative. I've attached a
> patch with an initial implementation of this. We've been running it on a few
> clusters, and have seen AM preemptions drop from double-digit occurrences on
> many days to zero.
> Of course, the tradeoff is some loss of locality, since the starved app is
> less likely to be allocated resources at the most specific locality level
> that it asked for. My opinion is that this tradeoff is worth it, but
> interested to hear what others think as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7903) Method getStarvedResourceRequests() only consider the first encountered resource
[
https://issues.apache.org/jira/browse/YARN-7903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357515#comment-16357515
]
Yufei Gu commented on YARN-7903:
{quote}
I think that we should try to make progress on that JIRA as well as this one.
{quote}
Agreed.
The request request in description is actually one RR instead of 3 RRs. If the
RR is strict about locality(RelaxLocality is false), I don't think there is
need to consider other RRs. However, that's another story if RR's RelaxLocality
is true. To enable some kind of delay scheduling for preemption seems a
reasonable solution for both YARN-6956 and this Jira.
I am still confusing about how to parse the multiple RRs of an apps, e.g. the
example in the description is actually one RRs instead of 3 RRs, what if there
size and container# are different between nodeRequest and rackRequest? Do we
consider them one RRs or multiple RRs. Without good understanding of this, I
don't think we can make any progress on YARN-6956. Please let me know your
thoughts about this. I'll try to dig more about this as well. Thanks. [~Steven
Rand].
> Method getStarvedResourceRequests() only consider the first encountered
> resource
>
>
> Key: YARN-7903
> URL: https://issues.apache.org/jira/browse/YARN-7903
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Priority: Major
>
> We need to specify rack and ANY while submitting a node local resource
> request, as YARN-7561 discussed. For example:
> {code}
> ResourceRequest nodeRequest =
> createResourceRequest(GB, node1.getHostName(), 1, 1, false);
> ResourceRequest rackRequest =
> createResourceRequest(GB, node1.getRackName(), 1, 1, false);
> ResourceRequest anyRequest =
> createResourceRequest(GB, ResourceRequest.ANY, 1, 1, false);
> List resourceRequests =
> Arrays.asList(nodeRequest, rackRequest, anyRequest);
> {code}
> However, method getStarvedResourceRequests() only consider the first
> encountered resource, which most likely is ResourceRequest.ANY. That's a
> mismatch for locality request.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357505#comment-16357505 ] Shane Kumpf commented on YARN-5428: --- Thanks [~jianhe] for the commit! > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf >Priority: Major > Labels: oct16-medium > Fix For: 3.1.0 > > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch, YARN-5428.004.patch, YARN-5428.005.patch, > YARN-5428.006.patch, YARN-5428.007.patch, YARN-5428.008.patch, > YARN-5428.009.patch, > YARN-5428Allowforspecifyingthedockerclientconfigurationdirectory.pdf > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357491#comment-16357491 ] Hudson commented on YARN-5428: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13635 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13635/]) YARN-5428. Allow for specifying the docker client configuration (jianhe: rev eb2449d5398e9ac869bc088e10d838a7f13deac0) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/DockerClientConfigHandler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/security/TestDockerClientConfigHandler.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/DockerCredentialTokenIdentifier.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/proto/yarn_security_token.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerRunCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf >Priority: Major > Labels: oct16-medium > Fix For: 3.1.0 > > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch, YARN-5428.004.patch, YARN-5428.005.patch, > YARN-5428.006.patch, YARN-5428.007.patch, YARN-5428.008.patch, > YARN-5428.009.patch, > YARN-5428Allowforspecifyingthedockerclientconfigurationdirectory.pdf > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7835) [Atsv2] Race condition in NM while publishing events if second attempt launched on same node
[ https://issues.apache.org/jira/browse/YARN-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357463#comment-16357463 ] Haibo Chen commented on YARN-7835: -- Thanks for the patch [~rohithsharma]. A few comments: 1) The new code in initializeContainer() and stopContainer() involves synchronized blocks, so I guess the code is meant to be thread safe. In initializeContainer(), if two threads see the container set for a given application is missing at the same time, they would create a singleton set respectively, but one would override the other. I think we could just synchronize on appIdToContainerId in both initializeContainer() and stopContainer() as we do with collectors in TimelineCollectorManager. 2) Not sure why we test `!auxService.hasApplication(appAttemptId.getApplicationId())` in a loop after the 1st attempt stopped. IIUC, the application should never be removed given the 2rd attempt is still running, so we'd just sleep all the time until the for loop counter goes over. Am I missing something? 3) For the second for-loop to wait for the application to be cleaned up, I think we could reuse GenericTestUtils.waitFor(). > [Atsv2] Race condition in NM while publishing events if second attempt > launched on same node > > > Key: YARN-7835 > URL: https://issues.apache.org/jira/browse/YARN-7835 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Critical > Attachments: YARN-7835.001.patch > > > It is observed race condition that if master container is killed for some > reason and launched on same node then NMTimelinePublisher doesn't add > timelineClient. But once completed container for 1st attempt has come then > NMTimelinePublisher removes the timelineClient. > It causes all subsequent event publishing from different client fails to > publish with exception Application is not found. ! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7530) hadoop-yarn-services-api should be part of hadoop-yarn-services
[ https://issues.apache.org/jira/browse/YARN-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reassigned YARN-7530: --- Assignee: Chandni Singh (was: Eric Yang) > hadoop-yarn-services-api should be part of hadoop-yarn-services > --- > > Key: YARN-7530 > URL: https://issues.apache.org/jira/browse/YARN-7530 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Chandni Singh >Priority: Trivial > Fix For: yarn-native-services > > Attachments: YARN-7530.001.patch > > > Hadoop-yarn-services-api is currently a parallel project to > hadoop-yarn-services project. It would be better if hadoop-yarn-services-api > is part of hadoop-yarn-services for correctness. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7813) Capacity Scheduler Intra-queue Preemption should be configurable for each queue
[
https://issues.apache.org/jira/browse/YARN-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357353#comment-16357353
]
Eric Payne commented on YARN-7813:
--
Thanks @jlowe for the comments!
I'm attaching a new patch that addresses your comments. It does not cherry-pick
cleanly to some of the previous branches. I am working on patches for those
other branches.
{quote}
Will this cause queues to start performing intra-queue preemption that did not
previously with the same configs or vice-versa?
{quote}
No. Previously, in-queue preemption was enabled and disabled at a particular
queue level via the cross-queue preemption config property. If nothing changes
in the configs, this behavior will remain the same. For example, if cross-queue
preemption is disabled at the root queue and then enabled at root.QueueA, all
children of QueueA will have both cross-queue and in-queue enabled. The
intra-queue preemption will only be disabled at a particular level or below if
the new property is set.
bq. "intreQueuePreemptionDisabled" s/b "intraQueuePreemptionDisabled"
Done.
{quote}
Why does CapacitySchedulerLeafQueueInfo have extra logic for getting
intra-queue preemption disabled status? I don't see this similar logic
elsewhere in the code.
{quote}
Yeah, I missed that. The desired behavior outlined above (if configs don't
change, intra-queue enablement doesn't change) was not working quite right, so
I moved some of the logic above the {{getIntraQueuePreemption}} and I didn't
make the change where it was really important
({{IntraQueueCandidatesSelector}}. I failed to do my usual rigorous testing so
I missed it. I rectified the problem by adding the
{{AbstractCSQueue#getIntraQueuePreemptionDisabledInHierarchy}} and then putting
the cross-queue <-> in-queue dependency logic in
{{AbstractCSQueue#getIntraQueuePreemptionDisabled}}.
bq. Technically the queue CLI output changes are incompatible per ...
Yup. I changed the output strings for preemption back to what they were before.
I like that better anyway ;-)
> Capacity Scheduler Intra-queue Preemption should be configurable for each
> queue
> ---
>
> Key: YARN-7813
> URL: https://issues.apache.org/jira/browse/YARN-7813
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.9.0, 2.8.3, 3.0.0
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
> Attachments: YARN-7813.001.patch, YARN-7813.002.patch
>
>
> Just as inter-queue (a.k.a. cross-queue) preemption is configurable per
> queue, intra-queue (a.k.a. in-queue) preemption should be configurable per
> queue. If a queue does not have a setting for intra-queue preemption, it
> should inherit its parents value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7813) Capacity Scheduler Intra-queue Preemption should be configurable for each queue
[ https://issues.apache.org/jira/browse/YARN-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-7813: - Attachment: YARN-7813.002.patch > Capacity Scheduler Intra-queue Preemption should be configurable for each > queue > --- > > Key: YARN-7813 > URL: https://issues.apache.org/jira/browse/YARN-7813 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, scheduler preemption >Affects Versions: 2.9.0, 2.8.3, 3.0.0 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Attachments: YARN-7813.001.patch, YARN-7813.002.patch > > > Just as inter-queue (a.k.a. cross-queue) preemption is configurable per > queue, intra-queue (a.k.a. in-queue) preemption should be configurable per > queue. If a queue does not have a setting for intra-queue preemption, it > should inherit its parents value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7626) Allow regular expression matching in container-executor.cfg for devices and named docker volumes mount
[ https://issues.apache.org/jira/browse/YARN-7626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357340#comment-16357340 ] Miklos Szegedi commented on YARN-7626: -- Thank you, [~Zian Chen] for the reply and update. Your latest patch does not seem to apply to trunk. Could you verify and rebase? > Allow regular expression matching in container-executor.cfg for devices and > named docker volumes mount > -- > > Key: YARN-7626 > URL: https://issues.apache.org/jira/browse/YARN-7626 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Zian Chen >Assignee: Zian Chen >Priority: Major > Attachments: YARN-7626.001.patch, YARN-7626.002.patch, > YARN-7626.003.patch, YARN-7626.004.patch, YARN-7626.005.patch, > YARN-7626.006.patch > > > Currently when we config some of the GPU devices related fields (like ) in > container-executor.cfg, these fields are generated based on different driver > versions or GPU device names. We want to enable regular expression matching > so that user don't need to manually set up these fields when config > container-executor.cfg, -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7836) YARN Service component update PUT API should not use component name from JSON body
[
https://issues.apache.org/jira/browse/YARN-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357301#comment-16357301
]
Billie Rinaldi commented on YARN-7836:
--
bq. if the JSON body contains a name attribute with value anything other than
the in the path param, we should send a 400 bad request saying they
do not match. If they are the same, it should be okay and we can process the
request.
This is analogous to our update service endpoint. In that case, we're ignoring
the name in the Service spec and overwriting it with the service name from the
path param. But I could see doing it differently for the component name
(sending a bad request as you're suggesting), since there is no reason for the
component names not to match.
> YARN Service component update PUT API should not use component name from JSON
> body
> --
>
> Key: YARN-7836
> URL: https://issues.apache.org/jira/browse/YARN-7836
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, yarn-native-services
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
>
> The YARN Service PUT API for component update should not use component name
> from the JSON body. The component update PUT URI is as follows -
> [http://localhost:9191/app/v1/services/]/components/
> e.g. [http://localhost:9191/app/v1/services/hello-world/components/hello]
> The component name is already in the URI, hence the JSON body expected should
> be only -
> {noformat}
> {
> "number_of_containers": 3
> }
> {noformat}
> It should not expect the name attribute in the JSON body. In fact, if the
> JSON body contains a name attribute with value anything other than the
> in the path param, we should send a 400 bad request saying they
> do not match. If they are the same, it should be okay and we can process the
> request.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7908) TestSystemMetricsPublisher#testPublishContainerMetrics can fail with an NPE
[
https://issues.apache.org/jira/browse/YARN-7908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357167#comment-16357167
]
Jason Lowe commented on YARN-7908:
--
The line that corresponds with that NPE is:
{code:java}
Assert.assertEquals(container.getAllocatedResource().getMemorySize(),
// KeyValueBasedTimelineStore could cast long to integer, need make sure
// variables for compare have same type.
((Integer) entity.getOtherInfo().get(
ContainerMetricsConstants.ALLOCATED_MEMORY_ENTITY_INFO))
.longValue());
{code}
Looks like the result of the {{get}} was null and the code blindly tried to
dereference it.
I think there may be issues with the state store persisting across multiple
tests. Many tests use the same container ID, and I think there could be a race
where the test polls for a container finished event not realizing this event is
a stale one from a previous test.
> TestSystemMetricsPublisher#testPublishContainerMetrics can fail with an NPE
> ---
>
> Key: YARN-7908
> URL: https://issues.apache.org/jira/browse/YARN-7908
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.3
>Reporter: Jason Lowe
>Priority: Major
>
> testPublishContainerMetrics can fail with a NullPointerException:
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
> Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.42 sec <<<
> FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
> testPublishContainerMetrics(org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher)
> Time elapsed: 0.031 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher.testPublishContainerMetrics(TestSystemMetricsPublisher.java:454)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7908) TestSystemMetricsPublisher#testPublishContainerMetrics can fail with an NPE
Jason Lowe created YARN-7908:
Summary: TestSystemMetricsPublisher#testPublishContainerMetrics
can fail with an NPE
Key: YARN-7908
URL: https://issues.apache.org/jira/browse/YARN-7908
Project: Hadoop YARN
Issue Type: Bug
Components: test
Affects Versions: 2.8.3
Reporter: Jason Lowe
testPublishContainerMetrics can fail with a NullPointerException:
{noformat}
Running
org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.42 sec <<<
FAILURE! - in
org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
testPublishContainerMetrics(org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher)
Time elapsed: 0.031 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher.testPublishContainerMetrics(TestSystemMetricsPublisher.java:454)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7907) Yarn app CLI client does not send Kerberos header to Resource Manager rest API
Yesha Vora created YARN-7907:
Summary: Yarn app CLI client does not send Kerberos header to
Resource Manager rest API
Key: YARN-7907
URL: https://issues.apache.org/jira/browse/YARN-7907
Project: Hadoop YARN
Issue Type: Bug
Components: yarn-native-services
Affects Versions: 3.0.0
Reporter: Yesha Vora
Launch of yarn service app is failing in secure mode with below stacktrace.
{code:java}
[hrt_qa@xxx root]$ kinit -kt
/home/hrt_qa/hadoopqa/keytabs/hrt_qa.headless.keytab hrt_qa
[hrt_qa@xxx root]$ yarn app -launch test2 sleeper
WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of
YARN_LOG_DIR.
WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of
YARN_LOGFILE.
WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of
YARN_PID_DIR.
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
18/02/07 22:50:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
18/02/07 22:50:41 INFO client.AHSProxy: Connecting to Application History
server at xxx/xxx:10200
18/02/07 22:50:41 INFO client.AHSProxy: Connecting to Application History
server at xxx/xxx:10200
18/02/07 22:50:41 INFO client.ApiServiceClient: Loading service definition from
local FS:
/usr/hdp/3.0.0.0-800/hadoop-yarn/yarn-service-examples/sleeper/sleeper.json
18/02/07 22:50:42 ERROR client.ApiServiceClient: Authentication required{code}
CLI client does not send Kerberos header to Resource Manager rest API. Tcpdump
indicate that there is no token being sent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7881) Add Log Aggregation Status API to the RM Webservice
[
https://issues.apache.org/jira/browse/YARN-7881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357011#comment-16357011
]
Gergely Novák commented on YARN-7881:
-
Thanks [~giovanni.fumarola] for your suggestion, I implemented it in patch #2.
> Add Log Aggregation Status API to the RM Webservice
> ---
>
> Key: YARN-7881
> URL: https://issues.apache.org/jira/browse/YARN-7881
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: yarn
>Reporter: Gergely Novák
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7881.001.patch, YARN-7881.002.patch
>
>
> The old YARN UI has a page: /cluster/logaggregationstatus/\{app_id} which
> shows the log aggregation status for all the nodes that run containers for
> the given application. In order to add a similar page to the new YARN UI we
> need to add an RM WS endpoint first.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7881) Add Log Aggregation Status API to the RM Webservice
[
https://issues.apache.org/jira/browse/YARN-7881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated YARN-7881:
Attachment: YARN-7881.002.patch
> Add Log Aggregation Status API to the RM Webservice
> ---
>
> Key: YARN-7881
> URL: https://issues.apache.org/jira/browse/YARN-7881
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: yarn
>Reporter: Gergely Novák
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7881.001.patch, YARN-7881.002.patch
>
>
> The old YARN UI has a page: /cluster/logaggregationstatus/\{app_id} which
> shows the log aggregation status for all the nodes that run containers for
> the given application. In order to add a similar page to the new YARN UI we
> need to add an RM WS endpoint first.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7875) AttributeStore for store and recover attributes
[ https://issues.apache.org/jira/browse/YARN-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-7875: --- Attachment: (was: YARN-7875-WIP.patch) > AttributeStore for store and recover attributes > --- > > Key: YARN-7875 > URL: https://issues.apache.org/jira/browse/YARN-7875 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-7875-WIP.patch > > > Similar to NodeLabelStore need to support NodeAttributeStore for persisting > attributes mapping to Nodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7875) AttributeStore for store and recover attributes
[
https://issues.apache.org/jira/browse/YARN-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356732#comment-16356732
]
Weiwei Yang commented on YARN-7875:
---
Hi [~bibinchundatt]
In {{FileSystemNodeLabelsStore}} and {{FileSystemNodeAttributeStore}}, can we
move
* close()
* initSchema()
* recover()
* setNodeLabelsManager() --> rename to setNodeDescriptorManager
to {{FileSystemDescriptorStore}} ?
Can you add some java doc to
{{FileSystemDescriptorStore#loadManagerFromEditLogStream}},
{{writeToMirrorFileStream}} and {{loadManagerFromMirror}} since they are the
major functions that requires each impl to override.
Thanks.
> AttributeStore for store and recover attributes
> ---
>
> Key: YARN-7875
> URL: https://issues.apache.org/jira/browse/YARN-7875
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-7875-WIP.patch, YARN-7875-WIP.patch
>
>
> Similar to NodeLabelStore need to support NodeAttributeStore for persisting
> attributes mapping to Nodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7827) Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
[
https://issues.apache.org/jira/browse/YARN-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356692#comment-16356692
]
Sunil G commented on YARN-7827:
---
Thanks [~eyang].
I changed hadoop.http.cross-origin.allowed-methods to add these methods in my
cluster while testing.
[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/HttpAuthentication.html]
mentions about this configuration in apache, so based on that i changed. I
dont think we need to change this doc, correct.
For UI perspective, we will add these information to the doc so that cluster
admin could take care of same.
> Stop and Delete Yarn Service from RM UI fails with HTTP ERROR 404
> -
>
> Key: YARN-7827
> URL: https://issues.apache.org/jira/browse/YARN-7827
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Yesha Vora
>Assignee: Sunil G
>Priority: Critical
> Attachments: Screen Shot 2018-02-07 at 3.51.54 PM.png,
> YARN-7827.001.patch, YARN-7827.002.patch
>
>
> Steps:
> 1) Enable Ats v2
> 2) Start Httpd Yarn service
> 3) Go to UI2 attempts page for yarn service
> 4) Click on setting icon
> 5) Click on stop service
> 6) This action will pop up a box to confirm stop. click on "Yes"
> Expected behavior:
> Yarn service should be stopped
> Actual behavior:
> Yarn UI is not notifying on whether Yarn service is stopped or not.
> On checking network stack trace, the PUT request failed with HTTP error 404
> {code}
> Sorry, got error 404
> Please consult RFC 2616 for meanings of the error code.
> Error Details
> org.apache.hadoop.yarn.webapp.WebAppException: /v1/services/httpd-hrt-qa-n:
> controller for v1 not found
> at org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:247)
> at org.apache.hadoop.yarn.webapp.Router.resolve(Router.java:155)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:143)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
> at
> com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1578)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
>
[jira] [Updated] (YARN-6462) Add yarn command to list all queues
[ https://issues.apache.org/jira/browse/YARN-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shen Yinjie updated YARN-6462: -- Description: we need a yarn command to list all leaf queues. (was: we need a yarn command to list all leaf queues. especially in some large scale cluster,there are many queues in tree-format, ) > Add yarn command to list all queues > --- > > Key: YARN-6462 > URL: https://issues.apache.org/jira/browse/YARN-6462 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Shen Yinjie >Assignee: Shen Yinjie >Priority: Major > Attachments: YARN-6462_2.patch > > > we need a yarn command to list all leaf queues. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6462) Add yarn command to list all queues
[ https://issues.apache.org/jira/browse/YARN-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shen Yinjie updated YARN-6462: -- Description: we need a yarn command to list all leaf queues. especially in some large scale cluster,there are many queues in tree-format, was:we need a yarn command to list all leaf queues, > Add yarn command to list all queues > --- > > Key: YARN-6462 > URL: https://issues.apache.org/jira/browse/YARN-6462 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Shen Yinjie >Assignee: Shen Yinjie >Priority: Major > Attachments: YARN-6462_2.patch > > > we need a yarn command to list all leaf queues. > especially in some large scale cluster,there are many queues in tree-format, -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6462) Add yarn command to list all queues
[ https://issues.apache.org/jira/browse/YARN-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shen Yinjie updated YARN-6462: -- Description: we need a yarn command to list all leaf queues, (was: we need a yarn command to list all queues ,and there is this kind of command for applications and nodemangers already... ) > Add yarn command to list all queues > --- > > Key: YARN-6462 > URL: https://issues.apache.org/jira/browse/YARN-6462 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Shen Yinjie >Assignee: Shen Yinjie >Priority: Major > Attachments: YARN-6462_2.patch > > > we need a yarn command to list all leaf queues, -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7875) AttributeStore for store and recover attributes
[
https://issues.apache.org/jira/browse/YARN-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356652#comment-16356652
]
Bibin A Chundatt edited comment on YARN-7875 at 2/8/18 8:51 AM:
[~cheersyang]
Thank you for looking at patch,
# Refactoring was done mainly to have {{CommonFileSystem}} implementation for
Store
# Read and write common code is pushed to {{FileSystemDescriptorStore}}
# Intention was to have 2 different store for Attributes and NodeLabels
{quote}
* updateNodeToLabelsMappings vs replaceNodeAttributes
* storeNewClusterNodeLabels vs addNodeAttributes
* removeClusterNodeLabels vs removeNodeAttributes
{quote}
addNodeAttributes,removeNodeAttributes,replaceNodeAttributes is for only Node
to Attributes mapping. storeNewClusterNodeLabels,removeClusterNodeLabels is
mainly for cluster level label addition
{quote}
If we really want to refactor the class to be more generic so labels and
attributes store can share most of code
{quote}
Could you explain further.
Along with above changes few modification related to mirror file opening is
also done as part of refactoring.
Will wait for [~sunil.gov...@gmail.com] and [~naganarasimha...@apache.org] too
before i upload first patch
was (Author: bibinchundatt):
[~cheersyang]
Thank you for looking at patch,
# Refactoring was done mainly to have CommonFileSystem implementation for
Store
# Currently i was planning to have 2 different store for Attributes and Labels
{quote}
* updateNodeToLabelsMappings vs replaceNodeAttributes
* storeNewClusterNodeLabels vs addNodeAttributes
* removeClusterNodeLabels vs removeNodeAttributes
{quote}
# addNodeAttributes,removeNodeAttributes,replaceNodeAttributes is for only
Node to Attributes mapping. storeNewClusterNodeLabels,removeClusterNodeLabels
is mainly for cluster level label addition
{quote}
{quote}
> AttributeStore for store and recover attributes
> ---
>
> Key: YARN-7875
> URL: https://issues.apache.org/jira/browse/YARN-7875
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-7875-WIP.patch, YARN-7875-WIP.patch
>
>
> Similar to NodeLabelStore need to support NodeAttributeStore for persisting
> attributes mapping to Nodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7875) AttributeStore for store and recover attributes
[
https://issues.apache.org/jira/browse/YARN-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356652#comment-16356652
]
Bibin A Chundatt commented on YARN-7875:
[~cheersyang]
Thank you for looking at patch,
# Refactoring was done mainly to have CommonFileSystem implementation for
Store
# Currently i was planning to have 2 different store for Attributes and Labels
{quote}
* updateNodeToLabelsMappings vs replaceNodeAttributes
* storeNewClusterNodeLabels vs addNodeAttributes
* removeClusterNodeLabels vs removeNodeAttributes
{quote}
# addNodeAttributes,removeNodeAttributes,replaceNodeAttributes is for only
Node to Attributes mapping. storeNewClusterNodeLabels,removeClusterNodeLabels
is mainly for cluster level label addition
{quote}
{quote}
> AttributeStore for store and recover attributes
> ---
>
> Key: YARN-7875
> URL: https://issues.apache.org/jira/browse/YARN-7875
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-7875-WIP.patch, YARN-7875-WIP.patch
>
>
> Similar to NodeLabelStore need to support NodeAttributeStore for persisting
> attributes mapping to Nodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7875) AttributeStore for store and recover attributes
[ https://issues.apache.org/jira/browse/YARN-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-7875: --- Attachment: YARN-7875-WIP.patch > AttributeStore for store and recover attributes > --- > > Key: YARN-7875 > URL: https://issues.apache.org/jira/browse/YARN-7875 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-7875-WIP.patch, YARN-7875-WIP.patch > > > Similar to NodeLabelStore need to support NodeAttributeStore for persisting > attributes mapping to Nodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
