[jira] [Updated] (YARN-4721) RM to try to auth with HDFS on startup, retry with max diagnostics on failure
[
https://issues.apache.org/jira/browse/YARN-4721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-4721:
---
Labels: oct16-medium (was: )
> RM to try to auth with HDFS on startup, retry with max diagnostics on failure
> -
>
> Key: YARN-4721
> URL: https://issues.apache.org/jira/browse/YARN-4721
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager, security
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Labels: oct16-medium
> Attachments: HADOOP-12289-002.patch, HADOOP-12289-003.patch,
> HADOOP-12889-001.patch
>
>
> If the RM can't auth with HDFS, this can first surface during job submission,
> which can cause confusion about what's wrong and whose credentials are
> playing up.
> Instead, the RM could try to talk to HDFS on launch, {{ls /}} should suffice.
> If it can't auth, it can then tell UGI to log more and retry.
> I don't know what the policy should be if the RM can't auth to HDFS at this
> point. Certainly it can't currently accept work. But should it fail fast or
> keep going in the hope that the problem is in the KDC or NN and will fix
> itself without an RM restart?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3933) Race condition when calling AbstractYarnScheduler.completedContainer.
[ https://issues.apache.org/jira/browse/YARN-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-3933: --- Labels: oct16-medium (was: ) > Race condition when calling AbstractYarnScheduler.completedContainer. > - > > Key: YARN-3933 > URL: https://issues.apache.org/jira/browse/YARN-3933 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.0, 2.7.0, 2.5.2, 2.7.1 >Reporter: Lavkesh Lahngir >Assignee: Shiwei Guo > Labels: oct16-medium > Attachments: YARN-3933.001.patch, YARN-3933.002.patch, > YARN-3933.003.patch, YARN-3933.004.patch, YARN-3933.005.patch > > > In our cluster we are seeing available memory and cores being negative. > Initial inspection: > Scenario no. 1: > In capacity scheduler the method allocateContainersToNode() checks if > there are excess reservation of containers for an application, and they are > no longer needed then it calls queue.completedContainer() which causes > resources being negative. And they were never assigned in the first place. > I am still looking through the code. Can somebody suggest how to simulate > excess containers assignments ? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5308) FairScheduler: Move continuous scheduling related tests to TestContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-5308: - Labels: newbie oct16-easy (was: newbie) > FairScheduler: Move continuous scheduling related tests to > TestContinuousScheduling > --- > > Key: YARN-5308 > URL: https://issues.apache.org/jira/browse/YARN-5308 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler, test >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Kai Sasaki > Labels: newbie, oct16-easy > Attachments: YARN-5308.01.patch, YARN-5308.02.patch, > YARN-5308.03.patch, YARN-5308.04.patch, YARN-5308.05.patch > > > TestFairScheduler still has some tests on continuous scheduling. We should > move them to TestContinuousScheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4721) RM to try to auth with HDFS on startup, retry with max diagnostics on failure
[
https://issues.apache.org/jira/browse/YARN-4721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-4721:
---
Component/s: security
> RM to try to auth with HDFS on startup, retry with max diagnostics on failure
> -
>
> Key: YARN-4721
> URL: https://issues.apache.org/jira/browse/YARN-4721
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager, security
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Labels: oct16-medium
> Attachments: HADOOP-12289-002.patch, HADOOP-12289-003.patch,
> HADOOP-12889-001.patch
>
>
> If the RM can't auth with HDFS, this can first surface during job submission,
> which can cause confusion about what's wrong and whose credentials are
> playing up.
> Instead, the RM could try to talk to HDFS on launch, {{ls /}} should suffice.
> If it can't auth, it can then tell UGI to log more and retry.
> I don't know what the policy should be if the RM can't auth to HDFS at this
> point. Certainly it can't currently accept work. But should it fail fast or
> keep going in the hope that the problem is in the KDC or NN and will fix
> itself without an RM restart?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5308) FairScheduler: Move continuous scheduling related tests to TestContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-5308: - Issue Type: Task (was: Improvement) > FairScheduler: Move continuous scheduling related tests to > TestContinuousScheduling > --- > > Key: YARN-5308 > URL: https://issues.apache.org/jira/browse/YARN-5308 > Project: Hadoop YARN > Issue Type: Task > Components: fairscheduler, test >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Kai Sasaki > Labels: newbie, oct16-easy > Attachments: YARN-5308.01.patch, YARN-5308.02.patch, > YARN-5308.03.patch, YARN-5308.04.patch, YARN-5308.05.patch > > > TestFairScheduler still has some tests on continuous scheduling. We should > move them to TestContinuousScheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4161) Capacity Scheduler : Assign single or multiple containers per heart beat driven by configuration
[ https://issues.apache.org/jira/browse/YARN-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated YARN-4161: Component/s: (was: capacityscheduler) capacity scheduler > Capacity Scheduler : Assign single or multiple containers per heart beat > driven by configuration > > > Key: YARN-4161 > URL: https://issues.apache.org/jira/browse/YARN-4161 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacity scheduler >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Labels: oct16-medium > Attachments: YARN-4161.patch > > > Capacity Scheduler right now schedules multiple containers per heart beat if > there are more resources available in the node. > This approach works fine however in some cases its not distribute the load > across the cluster hence throughput of the cluster suffers. I am adding > feature to drive that using configuration by that we can control the number > of containers assigned per heart beat. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3477) TimelineClientImpl swallows exceptions
[ https://issues.apache.org/jira/browse/YARN-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated YARN-3477: Labels: oct16-easy (was: ) > TimelineClientImpl swallows exceptions > -- > > Key: YARN-3477 > URL: https://issues.apache.org/jira/browse/YARN-3477 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0, 2.7.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Labels: oct16-easy > Attachments: YARN-3477-001.patch, YARN-3477-002.patch, > YARN-3477-trunk.003.patch, YARN-3477-trunk.004.patch > > > If timeline client fails more than the retry count, the original exception is > not thrown. Instead some runtime exception is raised saying "retries run out" > # the failing exception should be rethrown, ideally via > NetUtils.wrapException to include URL of the failing endpoing > # Otherwise, the raised RTE should (a) state that URL and (b) set the > original fault as the inner cause -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5714) ContainerExecutor does not order environment map
[ https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated YARN-5714: Labels: oct16-medium (was: ) > ContainerExecutor does not order environment map > > > Key: YARN-5714 > URL: https://issues.apache.org/jira/browse/YARN-5714 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1 > Environment: all (linux and windows alike) >Reporter: Remi Catherinot >Assignee: Remi Catherinot >Priority: Trivial > Labels: oct16-medium > Attachments: YARN-5714.001.patch, YARN-5714.002.patch, > YARN-5714.003.patch, YARN-5714.004.patch > > Original Estimate: 120h > Remaining Estimate: 120h > > when dumping the launch container script, environment variables are dumped > based on the order internally used by the map implementation (hash based). It > does not take into consideration that some env varibales may refer each > other, and so that some env variables must be declared before those > referencing them. > In my case, i ended up having LD_LIBRARY_PATH which was depending on > HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a > wrong value and so native libraries weren't loaded. jobs were running but not > at their best efficiency. This is just a use case falling into that bug, but > i'm sure others may happen as well. > I already have a patch running in my production environment, i just estimate > to 5 days for packaging the patch in the right fashion for JIRA + try my best > to add tests. > Note : the patch is not OS aware with a default empty implementation. I will > only implement the unix version on a 1st release. I'm not used to windows env > variables syntax so it will take me more time/research for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4597) Add SCHEDULE to NM container lifecycle
[ https://issues.apache.org/jira/browse/YARN-4597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-4597: Issue Type: New Feature (was: Bug) > Add SCHEDULE to NM container lifecycle > -- > > Key: YARN-4597 > URL: https://issues.apache.org/jira/browse/YARN-4597 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Reporter: Chris Douglas >Assignee: Arun Suresh > Attachments: YARN-4597.001.patch, YARN-4597.002.patch, > YARN-4597.003.patch, YARN-4597.004.patch, YARN-4597.005.patch > > > Currently, the NM immediately launches containers after resource > localization. Several features could be more cleanly implemented if the NM > included a separate stage for reserving resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5745) [YARN-3368] Disable war packaging under default profile for new YARN UI
[ https://issues.apache.org/jira/browse/YARN-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated YARN-5745: Labels: oct16-easy (was: ) > [YARN-3368] Disable war packaging under default profile for new YARN UI > --- > > Key: YARN-5745 > URL: https://issues.apache.org/jira/browse/YARN-5745 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: oct16-easy > Attachments: YARN-5745-YARN-3368.001.patch > > > Previously we enable war packaging for new UI project by default, which > causes all JS code including test code added to war package directly. We > should disable this behavior unless profile is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5736) YARN container executor config does not handle white space
[ https://issues.apache.org/jira/browse/YARN-5736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-5736: - Labels: oct16-medium (was: ) Component/s: nodemanager Issue Type: Improvement (was: Bug) > YARN container executor config does not handle white space > -- > > Key: YARN-5736 > URL: https://issues.apache.org/jira/browse/YARN-5736 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Trivial > Labels: oct16-medium > Attachments: YARN-5736.001.patch, YARN_5736.000.patch > > > The container executor configuration reader does not handle white spaces or > malformed key value pairs in the config file correctly or gracefully > as an example the following key value line which is part of the configuration > (note the << is used as a marker to show the extra trailing space): > yarn.nodemanager.linux-container-executor.group=yarn << > is a valid line but when you run the check over the file: > [root@test]#./container-executor --checksetup > Can't get group information for yarn - Success. > [root@test]# > It fails to find the yarn group but it really tries to find the "yarn " group > which fails. There is no trimming anywhere while processing the lines. If a > space would be added in before or after the = sign a failure would also occur. > Minor nit is the fact that a failure still is logged as a Success -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4532) Killing applications by appStates and queue in Yarn Application CLI
[
https://issues.apache.org/jira/browse/YARN-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Min Shen updated YARN-4532:
---
Labels: oct16-medium (was: )
> Killing applications by appStates and queue in Yarn Application CLI
> ---
>
> Key: YARN-4532
> URL: https://issues.apache.org/jira/browse/YARN-4532
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications, client
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: oct16-medium
> Attachments: YARN-4532.001.patch
>
>
> The cmd likes
> {code}
> -killByAppStates The states of application that will be killed.
> -killOfQueue Kill the applications of specific queue.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5267) RM REST API doc for app lists "Application Type" instead of "applicationType"
[
https://issues.apache.org/jira/browse/YARN-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Chiang updated YARN-5267:
-
Component/s: (was: api)
restapi
> RM REST API doc for app lists "Application Type" instead of "applicationType"
> --
>
> Key: YARN-5267
> URL: https://issues.apache.org/jira/browse/YARN-5267
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation, restapi
>Affects Versions: 2.6.4
>Reporter: Grant Sohn
>Priority: Trivial
> Labels: documentation, oct16-easy
> Attachments: YARN-5267.001.patch
>
>
> From the docs:
> {noformat}
> Note that depending on security settings a user might not be able to see all
> the fields.
> Item Data Type Description
> idstring The application id
> user string The user who started the application
> name string The application name
> Application Type string The application type
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4710) Reduce logging application reserved debug info in FSAppAttempt#assignContainer
[
https://issues.apache.org/jira/browse/YARN-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-4710:
---
Labels: oct16-easy (was: )
> Reduce logging application reserved debug info in FSAppAttempt#assignContainer
> --
>
> Key: YARN-4710
> URL: https://issues.apache.org/jira/browse/YARN-4710
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Labels: oct16-easy
> Attachments: YARN-4710.001.patch, yarn-debug.log
>
>
> I found lots of unimportant records info in assigning container when I
> prepared to debug the problem of container assigning.There are too many
> records like this in yarn-resourcemanager.log, and it's difficiult for me to
> directly to found the important info.
> {code}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:52,971 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:52,976 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:52,981 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:52,986 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:52,991 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:52,996 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:53,001 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:53,007 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:53,012 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:53,017 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:53,022 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:53,027 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:53,032 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:53,038 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:53,050 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> Node offered to app: application_1449458968698_0011 reserved: false
> 2016-02-21 16:31:53,057 DEBUG
> {code}
> The reason why of so many records is that it always print this info first in
> container assigning whether the assigned result is successful or failed.
> Can see the complete yarn log in updated log, and you can see how many
> records there are.
> And in addition, too many these info logging will slow down process of
> container assigning.Maybe we should change this logLevel to other level, like
> {{trace}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4703) blacklist option for distributedshell
[ https://issues.apache.org/jira/browse/YARN-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4703: --- Labels: oct16-easy (was: ) > blacklist option for distributedshell > -- > > Key: YARN-4703 > URL: https://issues.apache.org/jira/browse/YARN-4703 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications/distributed-shell >Affects Versions: 2.7.1 >Reporter: kyungwan nam >Assignee: kyungwan nam >Priority: Minor > Labels: oct16-easy > Attachments: YARN-4703.001.patch > > > we need to option “blacklist” in distributedshell. > It can be set so that the container is not allocated to specific nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5773) RM recovery too slow due to LeafQueue#activateApplication()
[
https://issues.apache.org/jira/browse/YARN-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612728#comment-15612728
]
Hadoop QA commented on YARN-5773:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 36m 20s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| | hadoop.yarn.server.resourcemanager.TestRMRestart |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5773 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835620/YARN-5773.0006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux cc52a21531b2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ac35ee9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/13558/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13558/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
[jira] [Updated] (YARN-3514) Active directory usernames like domain\login cause YARN failures
[ https://issues.apache.org/jira/browse/YARN-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated YARN-3514: Labels: oct16-easy (was: BB2015-05-TBR) > Active directory usernames like domain\login cause YARN failures > > > Key: YARN-3514 > URL: https://issues.apache.org/jira/browse/YARN-3514 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.2.0 > Environment: CentOS6 >Reporter: john lilley >Priority: Minor > Labels: oct16-easy > Attachments: YARN-3514.001.patch, YARN-3514.002.patch > > > We have a 2.2.0 (Cloudera 5.3) cluster running on CentOS6 that is > Kerberos-enabled and uses an external AD domain controller for the KDC. We > are able to authenticate, browse HDFS, etc. However, YARN fails during > localization because it seems to get confused by the presence of a \ > character in the local user name. > Our AD authentication on the nodes goes through sssd and set configured to > map AD users onto the form domain\username. For example, our test user has a > Kerberos principal of hadoopu...@domain.com and that maps onto a CentOS user > "domain\hadoopuser". We have no problem validating that user with PAM, > logging in as that user, su-ing to that user, etc. > However, when we attempt to run a YARN application master, the localization > step fails when setting up the local cache directory for the AM. The error > that comes out of the RM logs: > 2015-04-17 12:47:09 INFO net.redpoint.yarnapp.Client[0]: monitorApplication: > ApplicationReport: appId=1, state=FAILED, progress=0.0, finalStatus=FAILED, > diagnostics='Application application_1429295486450_0001 failed 1 times due to > AM Container for appattempt_1429295486450_0001_01 exited with exitCode: > -1000 due to: Application application_1429295486450_0001 initialization > failed (exitCode=255) with output: main : command provided 0 > main : user is DOMAIN\hadoopuser > main : requested yarn user is domain\hadoopuser > org.apache.hadoop.util.DiskChecker$DiskErrorException: Cannot create > directory: > /data/yarn/nm/usercache/domain%5Chadoopuser/appcache/application_1429295486450_0001/filecache/10 > at > org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:105) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.download(ContainerLocalizer.java:199) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:241) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:169) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:347) > .Failing this attempt.. Failing the application.' > However, when we look on the node launching the AM, we see this: > [root@rpb-cdh-kerb-2 ~]# cd /data/yarn/nm/usercache > [root@rpb-cdh-kerb-2 usercache]# ls -l > drwxr-s--- 4 DOMAIN\hadoopuser yarn 4096 Apr 17 12:10 domain\hadoopuser > There appears to be different treatment of the \ character in different > places. Something creates the directory as "domain\hadoopuser" but something > else later attempts to use it as "domain%5Chadoopuser". I’m not sure where > or why the URL escapement converts the \ to %5C or why this is not consistent. > I should also mention, for the sake of completeness, our auth_to_local rule > is set up to map u...@domain.com to domain\user: > RULE:[1:$1@$0](^.*@DOMAIN\.COM$)s/^(.*)@DOMAIN\.COM$/domain\\$1/g -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5745) [YARN-3368] Disable war packaging under default profile for new YARN UI
[ https://issues.apache.org/jira/browse/YARN-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated YARN-5745: Component/s: webapp > [YARN-3368] Disable war packaging under default profile for new YARN UI > --- > > Key: YARN-5745 > URL: https://issues.apache.org/jira/browse/YARN-5745 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: oct16-easy > Attachments: YARN-5745-YARN-3368.001.patch > > > Previously we enable war packaging for new UI project by default, which > causes all JS code including test code added to war package directly. We > should disable this behavior unless profile is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5774) MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set yarn.scheduler.minimum-allocation-mb to 0.
[
https://issues.apache.org/jira/browse/YARN-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated YARN-5774:
-
Labels: oct16-easy (was: )
> MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set
> yarn.scheduler.minimum-allocation-mb to 0.
>
>
> Key: YARN-5774
> URL: https://issues.apache.org/jira/browse/YARN-5774
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Labels: oct16-easy
> Attachments: YARN-5774.001.patch
>
>
> MR Job stuck in ACCEPTED status without any progress in Fair Scheduler
> because there is no resource request for the AM. This happened when you
> configure {{yarn.scheduler.minimum-allocation-mb}} to zero.
> The problem is in the code used by both Capacity Scheduler and Fair
> Scheduler. {{scheduler.increment-allocation-mb}} is a concept in FS, but not
> CS. So the common code in class RMAppManager passes the
> {{yarn.scheduler.minimum-allocation-mb}} as incremental one because there is
> no incremental one for CS when it tried to normalize the resource requests.
> {code}
> SchedulerUtils.normalizeRequest(amReq, scheduler.getResourceCalculator(),
> scheduler.getClusterResource(),
> scheduler.getMinimumResourceCapability(),
> scheduler.getMaximumResourceCapability(),
> scheduler.getMinimumResourceCapability()); --> incrementResource
> should be passed here.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1151) Ability to configure auxiliary services from HDFS-based JAR files
[ https://issues.apache.org/jira/browse/YARN-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612723#comment-15612723 ] Arun Suresh commented on YARN-1151: --- Have we reached consensus on the approach ? Canceling patch till then.. > Ability to configure auxiliary services from HDFS-based JAR files > - > > Key: YARN-1151 > URL: https://issues.apache.org/jira/browse/YARN-1151 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.1.0-beta, 2.9.0 >Reporter: john lilley >Assignee: Xuan Gong > Labels: auxiliary-service, yarn > Attachments: YARN-1151.1.patch > > > I would like to install an auxiliary service in Hadoop YARN without actually > installing files/services on every node in the system. Discussions on the > user@ list indicate that this is not easily done. The reason we want an > auxiliary service is that our application has some persistent-data components > that are not appropriate for HDFS. In fact, they are somewhat analogous to > the mapper output of MapReduce's shuffle, which is what led me to > auxiliary-services in the first place. It would be much easier if we could > just place our service's JARs in HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4862) Handle duplicate completed containers in RMNodeImpl
[
https://issues.apache.org/jira/browse/YARN-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612726#comment-15612726
]
Hadoop QA commented on YARN-4862:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 19s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 138 unchanged - 1 fixed = 139 total (was 139)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 38m
12s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-4862 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12814032/0003-YARN-4862.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fad3b211da0c 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ac35ee9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/13557/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/13557/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13557/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13557/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.or
[jira] [Updated] (YARN-5776) Checkstyle: MonitoringThread.Run method length is too long
[ https://issues.apache.org/jira/browse/YARN-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated YARN-5776: Component/s: nodemanager > Checkstyle: MonitoringThread.Run method length is too long > -- > > Key: YARN-5776 > URL: https://issues.apache.org/jira/browse/YARN-5776 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Trivial > Labels: oct16-easy > Attachments: YARN-5776.000.patch > > > YARN-5725 had a check style violation that should be resolved by refactoring > the function > Details: > ContainersMonitorImpl.java:395 MonitioringThread.Run @Override:5: Method > length is 233 lines (max allowed is 150). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5776) Checkstyle: MonitoringThread.Run method length is too long
[ https://issues.apache.org/jira/browse/YARN-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated YARN-5776: Labels: oct16-easy (was: ) > Checkstyle: MonitoringThread.Run method length is too long > -- > > Key: YARN-5776 > URL: https://issues.apache.org/jira/browse/YARN-5776 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Trivial > Labels: oct16-easy > Attachments: YARN-5776.000.patch > > > YARN-5725 had a check style violation that should be resolved by refactoring > the function > Details: > ContainersMonitorImpl.java:395 MonitioringThread.Run @Override:5: Method > length is 233 lines (max allowed is 150). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5267) RM REST API doc for app lists "Application Type" instead of "applicationType"
[
https://issues.apache.org/jira/browse/YARN-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Chiang updated YARN-5267:
-
Labels: documentation oct16-easy (was: documentation)
> RM REST API doc for app lists "Application Type" instead of "applicationType"
> --
>
> Key: YARN-5267
> URL: https://issues.apache.org/jira/browse/YARN-5267
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api, documentation
>Affects Versions: 2.6.4
>Reporter: Grant Sohn
>Priority: Trivial
> Labels: documentation, oct16-easy
> Attachments: YARN-5267.001.patch
>
>
> From the docs:
> {noformat}
> Note that depending on security settings a user might not be able to see all
> the fields.
> Item Data Type Description
> idstring The application id
> user string The user who started the application
> name string The application name
> Application Type string The application type
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4675) Reorganize TimeClientImpl into TimeClientV1Impl and TimeClientV2Impl
[ https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4675: --- Labels: YARN-5355 oct16-medium (was: YARN-5355) > Reorganize TimeClientImpl into TimeClientV1Impl and TimeClientV2Impl > > > Key: YARN-4675 > URL: https://issues.apache.org/jira/browse/YARN-4675 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Labels: YARN-5355, oct16-medium > Attachments: YARN-4675-YARN-2928.v1.001.patch > > > We need to reorganize TimeClientImpl into TimeClientV1Impl , > TimeClientV2Impl and if required a base class, so that its clear which part > of the code belongs to which version and thus better maintainable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4674) Allow RM REST API query apps by user list
[
https://issues.apache.org/jira/browse/YARN-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-4674:
---
Labels: oct16-easy (was: )
> Allow RM REST API query apps by user list
> -
>
> Key: YARN-4674
> URL: https://issues.apache.org/jira/browse/YARN-4674
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager, restapi
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Tao Jie
>Assignee: Tao Jie
>Priority: Minor
> Labels: oct16-easy
> Attachments: YARN-4674.001.patch, YARN-4674.002.patch
>
>
> Allow RM REST API query apps a list of users such as:
> {{http:///ws/v1/cluster/apps?users=user1,user2}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4674) Allow RM REST API query apps by user list
[
https://issues.apache.org/jira/browse/YARN-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-4674:
---
Component/s: resourcemanager
restapi
> Allow RM REST API query apps by user list
> -
>
> Key: YARN-4674
> URL: https://issues.apache.org/jira/browse/YARN-4674
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager, restapi
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Tao Jie
>Assignee: Tao Jie
>Priority: Minor
> Attachments: YARN-4674.001.patch, YARN-4674.002.patch
>
>
> Allow RM REST API query apps a list of users such as:
> {{http:///ws/v1/cluster/apps?users=user1,user2}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4529) Yarn CLI killing applications in batch
[
https://issues.apache.org/jira/browse/YARN-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Min Shen updated YARN-4529:
---
Labels: oct16-easy (was: )
> Yarn CLI killing applications in batch
> --
>
> Key: YARN-4529
> URL: https://issues.apache.org/jira/browse/YARN-4529
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: applications, client
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: oct16-easy
> Attachments: YARN-4529.001.patch
>
>
> We have not a good way to kill applications conveniently when starting some
> apps unexpected. At present, we have to kill them one by one. We can add some
> kill command that can kill apps in batch, like these:
> {code}
> -killByAppStatesThe states of application that will be killed.
> -killByUser Kill running-state applications of specific
> user.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3538) TimelineServer doesn't catch/translate all exceptions raised
[ https://issues.apache.org/jira/browse/YARN-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated YARN-3538: Labels: oct16-easy (was: BB2015-05-TBR) > TimelineServer doesn't catch/translate all exceptions raised > > > Key: YARN-3538 > URL: https://issues.apache.org/jira/browse/YARN-3538 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: oct16-easy > Attachments: YARN-3538-001.patch > > > Not all exceptions in TimelineServer are uprated to web exceptions; only IOEs -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5788) AM limit resource in UI and REST not updated after -replaceLabelsOnNode
[ https://issues.apache.org/jira/browse/YARN-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612701#comment-15612701 ] Bibin A Chundatt commented on YARN-5788: Yes we have to update the same on node labels update also.root.clusterResource should be able handle. > AM limit resource in UI and REST not updated after -replaceLabelsOnNode > --- > > Key: YARN-5788 > URL: https://issues.apache.org/jira/browse/YARN-5788 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: Screenshot from 2016-10-27 22-31-46.png > > > Steps to reproduce > == > # Enable node labels > # Configure capacity scheduler xml with label capacity configuration > # Add labelx to cluster > # Replace Node labels on node1 > # Check scheduler and Rest API > *Actual* > Am limit in scheduler UI is still based on old resource > *Expected* > AM limit to be updated based new partition resource -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5773) RM recovery too slow due to LeafQueue#activateApplication()
[
https://issues.apache.org/jira/browse/YARN-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612695#comment-15612695
]
Bibin A Chundatt commented on YARN-5773:
Attached patch removing num of apps check
> RM recovery too slow due to LeafQueue#activateApplication()
> ---
>
> Key: YARN-5773
> URL: https://issues.apache.org/jira/browse/YARN-5773
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-5773.0001.patch, YARN-5773.0002.patch,
> YARN-5773.0004.patch, YARN-5773.0005.patch, YARN-5773.0006.patch,
> YARN-5773.003.patch
>
>
> # Submit application 10K application to default queue.
> # All applications are in accepted state
> # Now restart resourcemanager
> For each application recovery {{LeafQueue#activateApplications()}} is
> invoked.Resulting in AM limit check to be done even before Node managers are
> getting registered.
> Total iteration for N application is about {{N(N+1)/2}} for {{10K}}
> application {{5000}} iterations causing time take for Rm to be active
> more than 10 min.
> Since NM resources are not yet added to during recovery we should skip
> {{activateApplicaiton()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5746) The state of the parentQueue and its childQueues should be synchronized.
[
https://issues.apache.org/jira/browse/YARN-5746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612688#comment-15612688
]
Daniel Templeton commented on YARN-5746:
Thanks, [~xgong]. Couple of comments:
* Let's collapse these nested conditionals into an _else if_: {code}}
else {
if (configuredState == QueueState.RUNNING
&& parentState == QueueState.STOPPED) {
throw new IllegalArgumentException(
"Illegal" + " State of " + configuredState
+ " for children of queue: " + queueName
+ ". The state of its parent queue: " + parent.getQueueName()
+ " is " + parentState);
} else {
this.state = configuredState;
}
}{code}
* It would be cleaner if {{getState()}} were rewritten to use
{{getConfiguredState()}}
> The state of the parentQueue and its childQueues should be synchronized.
>
>
> Key: YARN-5746
> URL: https://issues.apache.org/jira/browse/YARN-5746
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-5746.1.patch, YARN-5746.2.patch
>
>
> The state of the parentQueue and its childQeues need to be synchronized.
> * If the state of the parentQueue becomes STOPPED, the state of its
> childQueue need to become STOPPED as well.
> * If we change the state of the queue to RUNNING, we should make sure the
> state of all its ancestor must be RUNNING. Otherwise, we need to fail this
> operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4831) Recovered containers will be killed after NM stateful restart
[
https://issues.apache.org/jira/browse/YARN-4831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612685#comment-15612685
]
Jason Lowe commented on YARN-4831:
--
Sorry for the long delay in reviewing.
+1 lgtm. Will commit this pending a fresh Jenkins run.
> Recovered containers will be killed after NM stateful restart
> --
>
> Key: YARN-4831
> URL: https://issues.apache.org/jira/browse/YARN-4831
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siqi Li
>Assignee: Siqi Li
> Attachments: YARN-4831.v1.patch
>
>
> {code}
> 2016-03-04 19:43:48,130 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1456335621285_0040_01_66 transitioned from NEW to
> DONE
> 2016-03-04 19:43:48,130 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=henkins-service
>OPERATION=Container Finished - Killed TARGET=ContainerImpl
> RESULT=SUCCESS APPID=application_1456335621285_0040
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5788) AM limit resource in UI and REST not updated after -replaceLabelsOnNode
[ https://issues.apache.org/jira/browse/YARN-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612684#comment-15612684 ] Varun Saxena edited comment on YARN-5788 at 10/27/16 6:18 PM: -- Thanks [~bibinchundatt] for raising the issue. I guess this is because we calculate AM limits only while calling activating applications. So if no app comes, AM limit wont be updated. was (Author: varun_saxena): Thanks [~bibinchundatt] for raising the issue. I guess this is because we calculate AM limits only while calling activating applications. So if no app comes, resources wont be update. > AM limit resource in UI and REST not updated after -replaceLabelsOnNode > --- > > Key: YARN-5788 > URL: https://issues.apache.org/jira/browse/YARN-5788 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: Screenshot from 2016-10-27 22-31-46.png > > > Steps to reproduce > == > # Enable node labels > # Configure capacity scheduler xml with label capacity configuration > # Add labelx to cluster > # Replace Node labels on node1 > # Check scheduler and Rest API > *Actual* > Am limit in scheduler UI is still based on old resource > *Expected* > AM limit to be updated based new partition resource -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5788) AM limit resource in UI and REST not updated after -replaceLabelsOnNode
[ https://issues.apache.org/jira/browse/YARN-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612684#comment-15612684 ] Varun Saxena commented on YARN-5788: Thanks [~bibinchundatt] for raising the issue. I guess this is because we calculate AM limits only while calling activating applications. So if no app comes, resources wont be update. > AM limit resource in UI and REST not updated after -replaceLabelsOnNode > --- > > Key: YARN-5788 > URL: https://issues.apache.org/jira/browse/YARN-5788 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: Screenshot from 2016-10-27 22-31-46.png > > > Steps to reproduce > == > # Enable node labels > # Configure capacity scheduler xml with label capacity configuration > # Add labelx to cluster > # Replace Node labels on node1 > # Check scheduler and Rest API > *Actual* > Am limit in scheduler UI is still based on old resource > *Expected* > AM limit to be updated based new partition resource -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5545) App submit failure on queue with label when default queue partition capacity is zero
[
https://issues.apache.org/jira/browse/YARN-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612660#comment-15612660
]
Naganarasimha G R commented on YARN-5545:
-
All other things are fine just for the caveat that you mentioned.
bq. But One case to point out here is that, is there any use case by which
customer is expecting to divide max-apps by capacity (no queue override) and do
not want a default global max-apps?. If so, then we can add few more tuning
configs to forcefully enable capacity division of max-apps in each queue level
over default global max-app. Does this make sense?
So based on the consensus from [~jlowe],[~sunilg] & [~wangda], may be we can
conclude to currently go ahead with the same approach and if required to add
additional configs in future.
> App submit failure on queue with label when default queue partition capacity
> is zero
>
>
> Key: YARN-5545
> URL: https://issues.apache.org/jira/browse/YARN-5545
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-5545.0001.patch, YARN-5545.0002.patch,
> YARN-5545.0003.patch, YARN-5545.004.patch, capacity-scheduler.xml
>
>
> Configure capacity scheduler
> yarn.scheduler.capacity.root.default.capacity=0
> yarn.scheduler.capacity.root.queue1.accessible-node-labels.labelx.capacity=50
> yarn.scheduler.capacity.root.default.accessible-node-labels.labelx.capacity=50
> Submit application as below
> ./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha2-SNAPSHOT-tests.jar
> sleep -Dmapreduce.job.node-label-expression=labelx
> -Dmapreduce.job.queuename=default -m 1 -r 1 -mt 1000 -rt 1
> {noformat}
> 2016-08-21 18:21:31,375 INFO mapreduce.JobSubmitter: Cleaning up the staging
> area /tmp/hadoop-yarn/staging/root/.staging/job_1471670113386_0001
> java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed
> to submit application_1471670113386_0001 to YARN :
> org.apache.hadoop.security.AccessControlException: Queue root.default already
> has 0 applications, cannot accept submission of application:
> application_1471670113386_0001
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:316)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:255)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1790)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
> at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:273)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:136)
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:144)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
> application_1471670113386_0001 to YARN :
> org.apache.hadoop.security.AccessControlException: Queue root.default already
> has 0 applications, cannot accept submission of application:
> application_1471670113386_0001
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:286)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:296)
> at org.apache.hadoop.mapred.YARNRunner.submitJo
[jira] [Commented] (YARN-4907) Make all MockRM#waitForState consistent.
[
https://issues.apache.org/jira/browse/YARN-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612655#comment-15612655
]
Yufei Gu commented on YARN-4907:
Hi [~miklos.szeg...@cloudera.com], after looking at the
BaseContainerManagerTest.waitForNMContainerState closely, I think better to put
it in another JIRA.
> Make all MockRM#waitForState consistent.
> -
>
> Key: YARN-4907
> URL: https://issues.apache.org/jira/browse/YARN-4907
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-4907.001.patch
>
>
> There are some inconsistencies among these {{waitForState}} in {{MockRM}}:
> 1. Some {{waitForState}} return a boolean while others don't.
> 2. Some {{waitForState}} don't have a timeout, they can wait for ever.
> 3. Some {{waitForState}} use LOG.info and others use {{System.out.println}}
> to print messages.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4498) Application level node labels stats to be available in UI/REST/CLI
[ https://issues.apache.org/jira/browse/YARN-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612647#comment-15612647 ] Naganarasimha G R commented on YARN-4498: - Thanks [~bibinchundatt] for the patch, hope you can segregate the UI, REST and CLI into 2 or more patches and also capture some test case for the same. XML Out put looks fine enough ! > Application level node labels stats to be available in UI/REST/CLI > -- > > Key: YARN-4498 > URL: https://issues.apache.org/jira/browse/YARN-4498 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-4498.patch, YARN-4498.0002.patch, apps.xml > > > Currently nodelabel stats per application is not available through > REST/CLI/UI like currently used labels by all live containers, total stats of > containers per label for app etc.. > Will split into multiple task if approach is fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4218) Metric for resource*time that was preempted
[ https://issues.apache.org/jira/browse/YARN-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-4218: - Component/s: resourcemanager > Metric for resource*time that was preempted > --- > > Key: YARN-4218 > URL: https://issues.apache.org/jira/browse/YARN-4218 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Chang Li >Assignee: Chang Li > Attachments: YARN-4218.2.patch, YARN-4218.2.patch, YARN-4218.2.patch, > YARN-4218.2.patch, YARN-4218.3.patch, YARN-4218.patch, YARN-4218.wip.patch, > screenshot-1.png, screenshot-2.png, screenshot-3.png > > > After YARN-415 we have the ability to track the resource*time footprint of a > job and preemption metrics shows how many containers were preempted on a job. > However we don't have a metric showing the resource*time footprint cost of > preemption. In other words, we know how many containers were preempted but we > don't have a good measure of how much work was lost as a result of preemption. > We should add this metric so we can analyze how much work preemption is > costing on a grid and better track which jobs were heavily impacted by it. A > job that has 100 containers preempted that only lasted a minute each and were > very small is going to be less impacted than a job that only lost a single > container but that container was huge and had been running for 3 days. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5767) Fix the order that resources are cleaned up from the local Public/Private caches
[
https://issues.apache.org/jira/browse/YARN-5767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612611#comment-15612611
]
Sangjin Lee commented on YARN-5767:
---
+1 too.
> Fix the order that resources are cleaned up from the local Public/Private
> caches
>
>
> Key: YARN-5767
> URL: https://issues.apache.org/jira/browse/YARN-5767
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.0, 3.0.0-alpha1
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Attachments: YARN-5767-trunk-v1.patch, YARN-5767-trunk-v2.patch,
> YARN-5767-trunk-v3.patch, YARN-5767-trunk-v4.patch
>
>
> If you look at {{ResourceLocalizationService#handleCacheCleanup}}, you can
> see that public resources are added to the {{ResourceRetentionSet}} first
> followed by private resources:
> {code:java}
> private void handleCacheCleanup(LocalizationEvent event) {
> ResourceRetentionSet retain =
> new ResourceRetentionSet(delService, cacheTargetSize);
> retain.addResources(publicRsrc);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Resource cleanup (public) " + retain);
> }
> for (LocalResourcesTracker t : privateRsrc.values()) {
> retain.addResources(t);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Resource cleanup " + t.getUser() + ":" + retain);
> }
> }
> //TODO Check if appRsrcs should also be added to the retention set.
> }
> {code}
> Unfortunately, if we look at {{ResourceRetentionSet#addResources}} we see
> that this means public resources are deleted first until the target cache
> size is met:
> {code:java}
> public void addResources(LocalResourcesTracker newTracker) {
> for (LocalizedResource resource : newTracker) {
> currentSize += resource.getSize();
> if (resource.getRefCount() > 0) {
> // always retain resources in use
> continue;
> }
> retain.put(resource, newTracker);
> }
> for (Iterator> i =
> retain.entrySet().iterator();
>currentSize - delSize > targetSize && i.hasNext();) {
> Map.Entry rsrc = i.next();
> LocalizedResource resource = rsrc.getKey();
> LocalResourcesTracker tracker = rsrc.getValue();
> if (tracker.remove(resource, delService)) {
> delSize += resource.getSize();
> i.remove();
> }
> }
> }
> {code}
> The result of this is that resources in the private cache are only deleted in
> the cases where:
> # The cache size is larger than the target cache size and the public cache is
> empty.
> # The cache size is larger than the target cache size and everything in the
> public cache is being used by a running container.
> For clusters that primarily use the public cache (i.e. make use of the shared
> cache), this means that the most commonly used resources can be deleted
> before old resources in the private cache. Furthermore, the private cache can
> continue to grow over time causing more and more churn in the public cache.
> Additionally, the same problem exists within the private cache. Since
> resources are added to the retention set on a user by user basis, resources
> will get cleaned up one user at a time in the order that privateRsrc.values()
> returns the LocalResourcesTracker. So if user1 has 10MB in their cache and
> user2 has 100MB in their cache and the target size of the cache is 50MB,
> user1 could potentially have their entire cache removed before anything is
> deleted from the user2 cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4734) Merge branch:YARN-3368 to trunk
[ https://issues.apache.org/jira/browse/YARN-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612604#comment-15612604 ] Wangda Tan commented on YARN-4734: -- Sounds good :), any comments before I restart the vote? Thanks > Merge branch:YARN-3368 to trunk > --- > > Key: YARN-4734 > URL: https://issues.apache.org/jira/browse/YARN-4734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-4734.1.patch, YARN-4734.10-NOT_READY.patch, > YARN-4734.11-NOT_READY.patch, YARN-4734.12-NOT_READY.patch, > YARN-4734.13.patch, YARN-4734.14.patch, YARN-4734.15.patch, > YARN-4734.16.patch, YARN-4734.17-rebased.patch, YARN-4734.17.patch, > YARN-4734.2.patch, YARN-4734.3.patch, YARN-4734.4.patch, YARN-4734.5.patch, > YARN-4734.6.patch, YARN-4734.7.patch, YARN-4734.8.patch, > YARN-4734.9-NOT_READY.patch > > > YARN-2928 branch is planned to merge back to trunk shortly, it depends on > changes of YARN-3368. This JIRA is to track the merging task. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5788) AM limit resource in UI and REST not updated after -replaceLabelsOnNode
[ https://issues.apache.org/jira/browse/YARN-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-5788: --- Summary: AM limit resource in UI and REST not updated after -replaceLabelsOnNode (was: AM limit resource in UI and REST not updated after -replaceNode) > AM limit resource in UI and REST not updated after -replaceLabelsOnNode > --- > > Key: YARN-5788 > URL: https://issues.apache.org/jira/browse/YARN-5788 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: Screenshot from 2016-10-27 22-31-46.png > > > Steps to reproduce > == > # Enable node labels > # Configure capacity scheduler xml with label capacity configuration > # Add labelx to cluster > # Replace Node labels on node1 > # Check scheduler and Rest API > *Actual* > Am limit in scheduler UI is still based on old resource > *Expected* > AM limit to be updated based new partition resource -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5788) AM limit resource in UI and REST not updated after -replaceLabelsOnNode
[ https://issues.apache.org/jira/browse/YARN-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-5788: --- Attachment: Screenshot from 2016-10-27 22-31-46.png > AM limit resource in UI and REST not updated after -replaceLabelsOnNode > --- > > Key: YARN-5788 > URL: https://issues.apache.org/jira/browse/YARN-5788 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: Screenshot from 2016-10-27 22-31-46.png > > > Steps to reproduce > == > # Enable node labels > # Configure capacity scheduler xml with label capacity configuration > # Add labelx to cluster > # Replace Node labels on node1 > # Check scheduler and Rest API > *Actual* > Am limit in scheduler UI is still based on old resource > *Expected* > AM limit to be updated based new partition resource -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5788) AM limit resource in UI and REST not updated after -replaceNode
Bibin A Chundatt created YARN-5788: -- Summary: AM limit resource in UI and REST not updated after -replaceNode Key: YARN-5788 URL: https://issues.apache.org/jira/browse/YARN-5788 Project: Hadoop YARN Issue Type: Bug Reporter: Bibin A Chundatt Assignee: Bibin A Chundatt Priority: Minor Steps to reproduce == # Enable node labels # Configure capacity scheduler xml with label capacity configuration # Add labelx to cluster # Replace Node labels on node1 # Check scheduler and Rest API *Actual* Am limit in scheduler UI is still based on old resource *Expected* AM limit to be updated based new partition resource -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4862) Handle duplicate completed containers in RMNodeImpl
[ https://issues.apache.org/jira/browse/YARN-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612520#comment-15612520 ] Jason Lowe commented on YARN-4862: -- +1 latest patch lgtm. I can cleanup the whitespace and checkstyle nits on commit. It doesn't look like this change is ready to go even without YARN-5279, [~rohithsharma] please confirm. > Handle duplicate completed containers in RMNodeImpl > --- > > Key: YARN-4862 > URL: https://issues.apache.org/jira/browse/YARN-4862 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4862.patch, 0002-YARN-4862.patch, > 0003-YARN-4862.patch > > > As per > [comment|https://issues.apache.org/jira/browse/YARN-4852?focusedCommentId=15209689&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15209689] > from [~sharadag], there should be safe guard for duplicated container status > in RMNodeImpl before creating UpdatedContainerInfo. > Or else in heavily loaded cluster where event processing is gradually slow, > if any duplicated container are sent to RM(may be bug in NM also), there is > significant impact that RMNodImpl always create UpdatedContainerInfo for > duplicated containers. This result in increase in the heap memory and causes > problem like YARN-4852. > This is an optimization for issue kind YARN-4852 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4734) Merge branch:YARN-3368 to trunk
[ https://issues.apache.org/jira/browse/YARN-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612521#comment-15612521 ] Allen Wittenauer commented on YARN-4734: bq. One question: does this mean in the future all the precommit will run on latest Yetus master? Probably obvious from the above output (-SNAPSHOT in the version), but until configured otherwise, YARN precommit will now use Yetus master. > Merge branch:YARN-3368 to trunk > --- > > Key: YARN-4734 > URL: https://issues.apache.org/jira/browse/YARN-4734 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-4734.1.patch, YARN-4734.10-NOT_READY.patch, > YARN-4734.11-NOT_READY.patch, YARN-4734.12-NOT_READY.patch, > YARN-4734.13.patch, YARN-4734.14.patch, YARN-4734.15.patch, > YARN-4734.16.patch, YARN-4734.17-rebased.patch, YARN-4734.17.patch, > YARN-4734.2.patch, YARN-4734.3.patch, YARN-4734.4.patch, YARN-4734.5.patch, > YARN-4734.6.patch, YARN-4734.7.patch, YARN-4734.8.patch, > YARN-4734.9-NOT_READY.patch > > > YARN-2928 branch is planned to merge back to trunk shortly, it depends on > changes of YARN-3368. This JIRA is to track the merging task. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612519#comment-15612519 ] Haibo Chen commented on YARN-1593: -- [~djp] Any update? > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Junping Du > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4862) Handle duplicate completed containers in RMNodeImpl
[
https://issues.apache.org/jira/browse/YARN-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612479#comment-15612479
]
Hadoop QA commented on YARN-4862:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 19s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 138 unchanged - 1 fixed = 139 total (was 139)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 34m
49s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-4862 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12814032/0003-YARN-4862.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 163150d3e6a9 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ac35ee9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/13556/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/13556/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13556/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13556/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.or
[jira] [Updated] (YARN-5773) RM recovery too slow due to LeafQueue#activateApplication()
[
https://issues.apache.org/jira/browse/YARN-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-5773:
---
Attachment: YARN-5773.0006.patch
> RM recovery too slow due to LeafQueue#activateApplication()
> ---
>
> Key: YARN-5773
> URL: https://issues.apache.org/jira/browse/YARN-5773
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-5773.0001.patch, YARN-5773.0002.patch,
> YARN-5773.0004.patch, YARN-5773.0005.patch, YARN-5773.0006.patch,
> YARN-5773.003.patch
>
>
> # Submit application 10K application to default queue.
> # All applications are in accepted state
> # Now restart resourcemanager
> For each application recovery {{LeafQueue#activateApplications()}} is
> invoked.Resulting in AM limit check to be done even before Node managers are
> getting registered.
> Total iteration for N application is about {{N(N+1)/2}} for {{10K}}
> application {{5000}} iterations causing time take for Rm to be active
> more than 10 min.
> Since NM resources are not yet added to during recovery we should skip
> {{activateApplicaiton()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5773) RM recovery too slow due to LeafQueue#activateApplication()
[
https://issues.apache.org/jira/browse/YARN-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612443#comment-15612443
]
Naganarasimha G R commented on YARN-5773:
-
I too agree to not to have just 1 app to be in activated state
> RM recovery too slow due to LeafQueue#activateApplication()
> ---
>
> Key: YARN-5773
> URL: https://issues.apache.org/jira/browse/YARN-5773
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-5773.0001.patch, YARN-5773.0002.patch,
> YARN-5773.0004.patch, YARN-5773.0005.patch, YARN-5773.003.patch
>
>
> # Submit application 10K application to default queue.
> # All applications are in accepted state
> # Now restart resourcemanager
> For each application recovery {{LeafQueue#activateApplications()}} is
> invoked.Resulting in AM limit check to be done even before Node managers are
> getting registered.
> Total iteration for N application is about {{N(N+1)/2}} for {{10K}}
> application {{5000}} iterations causing time take for Rm to be active
> more than 10 min.
> Since NM resources are not yet added to during recovery we should skip
> {{activateApplicaiton()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5734) OrgQueue for easy CapacityScheduler queue configuration management
[ https://issues.apache.org/jira/browse/YARN-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612431#comment-15612431 ] Jonathan Hung commented on YARN-5734: - Here are the notes from yesterday's meetup: Objective: Aligning queue configuration requirements from YARN-5734 and YARN-5724 Attendees: Xuan, Wangda, Vinod, Subru, Zhe, Konstantin, Ye, Min, Jonathan, Erik 10/26/16 2-4pm Meeting minutes * Overall we are in agreement of adding a Mutable API for queue configuration. We discussed many details around APIs and storage implementation. * APIs ** For compatibility we can keep xml-file-based configuration as an option. Subru and Wangda both raised a concern that having 2 sources of truth is hard to maintain; therefore user should choose to use either the xml-file-based configuration approach or the new API-based one. ** Vinod raised a point that besides REST APIs, CLIs are also important. ** We also discussed a tricky case of adding new resources to the entire system. ** We discussed an advanced feature of supporting multi-update transactions. E.g. reducing capacity of queue A and moving the capacity of queue B. ** We discussed how to support bulk updates. ** We discussed how to make the project applicable for both Capacity and Fair schedulers. YARN-2986 should be revisited to provide a common data model for both schedulers. ** We discussed the case of hierarchical queues. * Storage implementation ** Derby embedded database can be used as default underlying storage implementation ** Storage implementation should be configurable, e.g. need to use distributed storage to support HA ** Another option is to use the YARN RM state store. This potentially simplifies how update events are logged (audit logger) and recovered. ** Need to address other issues, such as scheduler-agnostic REST APIs and user-friendly concurrent updates ** Target branch-2 * Action items ** Combine YARN-5724 and YARN-5734 to one umbrella ** Create one unified design doc covering *** Backing store implementations *** Queue state machine *** List of supported APIs ** Create feature branch (and add Min Shen ms...@linkedin.com, Jonathan Hung jyhung2...@gmail.com, Ye Zhou zhouye...@gmail.com as branch committers) ** After feature branch is created, create sub-tasks needed for implementing mutable API configuration provider > OrgQueue for easy CapacityScheduler queue configuration management > -- > > Key: YARN-5734 > URL: https://issues.apache.org/jira/browse/YARN-5734 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Min Shen >Assignee: Min Shen > Attachments: OrgQueue_Design_v0.pdf > > > The current xml based configuration mechanism in CapacityScheduler makes it > very inconvenient to apply any changes to the queue configurations. We saw 2 > main drawbacks in the file based configuration mechanism: > # This makes it very inconvenient to automate queue configuration updates. > For example, in our cluster setup, we leverage the queue mapping feature from > YARN-2411 to route users to their dedicated organization queues. It could be > extremely cumbersome to keep updating the config file to manage the very > dynamic mapping between users to organizations. > # Even a user has the admin permission on one specific queue, that user is > unable to make any queue configuration changes to resize the subqueues, > changing queue ACLs, or creating new queues. All these operations need to be > performed in a centralized manner by the cluster administrators. > With these current limitations, we realized the need of a more flexible > configuration mechanism that allows queue configurations to be stored and > managed more dynamically. We developed the feature internally at LinkedIn > which introduces the concept of MutableConfigurationProvider. What it > essentially does is to provide a set of configuration mutation APIs that > allows queue configurations to be updated externally with a set of REST APIs. > When performing the queue configuration changes, the queue ACLs will be > honored, which means only queue administrators can make configuration changes > to a given queue. MutableConfigurationProvider is implemented as a pluggable > interface, and we have one implementation of this interface which is based on > Derby embedded database. > This feature has been deployed at LinkedIn's Hadoop cluster for a year now, > and have gone through several iterations of gathering feedbacks from users > and improving accordingly. With this feature, cluster administrators are able > to automate lots of thequeue configuration management tasks, such as setting > the queue capacities to adjust cluster resources between queues based on > established resource consumption patterns, or managing
[jira] [Comment Edited] (YARN-4218) Metric for resource*time that was preempted
[ https://issues.apache.org/jira/browse/YARN-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612427#comment-15612427 ] Eric Payne edited comment on YARN-4218 at 10/27/16 4:32 PM: Cancelling patch since it no longer applies. I will review the patch and provide feedback shortly. was (Author: eepayne): Cancelling patch since it no longer applies. I will > Metric for resource*time that was preempted > --- > > Key: YARN-4218 > URL: https://issues.apache.org/jira/browse/YARN-4218 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: YARN-4218.2.patch, YARN-4218.2.patch, YARN-4218.2.patch, > YARN-4218.2.patch, YARN-4218.3.patch, YARN-4218.patch, YARN-4218.wip.patch, > screenshot-1.png, screenshot-2.png, screenshot-3.png > > > After YARN-415 we have the ability to track the resource*time footprint of a > job and preemption metrics shows how many containers were preempted on a job. > However we don't have a metric showing the resource*time footprint cost of > preemption. In other words, we know how many containers were preempted but we > don't have a good measure of how much work was lost as a result of preemption. > We should add this metric so we can analyze how much work preemption is > costing on a grid and better track which jobs were heavily impacted by it. A > job that has 100 containers preempted that only lasted a minute each and were > very small is going to be less impacted than a job that only lost a single > container but that container was huge and had been running for 3 days. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5773) RM recovery too slow due to LeafQueue#activateApplication()
[
https://issues.apache.org/jira/browse/YARN-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612408#comment-15612408
]
Varun Saxena edited comment on YARN-5773 at 10/27/16 4:26 PM:
--
bq. I did verify the same when cluster resource is empty and submit
application.The first application attempt is activated.
Yes, that will happen right now because we are checking for am limits inside
loop. But its not required in my opinion when we check for overall cluster
resources. Its not that its a bug per say in your current patch, its just that
condition in my opinion is unnecessary. It is just meant to be there for some
other reason.
As the log suggests it is kept there to cover cases where maximum am resource
percent is kept too low for a queue. We do not want to block apps in this case.
When overall cluster resources are 0, not even 1 application being activated is
because cluster resources are 0, not because am resource percent is
insufficient.
Thoughts ?
{code}
LOG.warn("maximum-am-resource-percent is insufficient to start a"
+ " single application in queue, it is likely set too low."
+ " skipping enforcement to allow at least one application"
+ " to start");
{code}
was (Author: varun_saxena):
bq. I did verify the same when cluster resource is empty and submit
application.The first application attempt is activated.
Yes, that will happen right now. But its not required in my opinion. Its not a
bug per say. It is just meant to be there for some other reason.
As the log suggests it is kept there to cover cases where maximum am resource
percent is kept too low for a queue. We do not want to block apps in this case.
When overall cluster resources are 0, not even 1 application being activated is
because cluster resources are 0, not because am resource percent is
insufficient.
Thoughts ?
{code}
LOG.warn("maximum-am-resource-percent is insufficient to start a"
+ " single application in queue, it is likely set too low."
+ " skipping enforcement to allow at least one application"
+ " to start");
{code}
> RM recovery too slow due to LeafQueue#activateApplication()
> ---
>
> Key: YARN-5773
> URL: https://issues.apache.org/jira/browse/YARN-5773
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-5773.0001.patch, YARN-5773.0002.patch,
> YARN-5773.0004.patch, YARN-5773.0005.patch, YARN-5773.003.patch
>
>
> # Submit application 10K application to default queue.
> # All applications are in accepted state
> # Now restart resourcemanager
> For each application recovery {{LeafQueue#activateApplications()}} is
> invoked.Resulting in AM limit check to be done even before Node managers are
> getting registered.
> Total iteration for N application is about {{N(N+1)/2}} for {{10K}}
> application {{5000}} iterations causing time take for Rm to be active
> more than 10 min.
> Since NM resources are not yet added to during recovery we should skip
> {{activateApplicaiton()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5773) RM recovery too slow due to LeafQueue#activateApplication()
[
https://issues.apache.org/jira/browse/YARN-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612408#comment-15612408
]
Varun Saxena commented on YARN-5773:
bq. I did verify the same when cluster resource is empty and submit
application.The first application attempt is activated.
Yes, that will happen right now. But its not required in my opinion. Its not a
bug per say. It is just meant to be there for some other reason.
As the log suggests it is kept there to cover cases where maximum am resource
percent is kept too low for a queue. We do not want to block apps in this case.
When overall cluster resources are 0, not even 1 application being activated is
because cluster resources are 0, not because am resource percent is
insufficient.
Thoughts ?
{code}
LOG.warn("maximum-am-resource-percent is insufficient to start a"
+ " single application in queue, it is likely set too low."
+ " skipping enforcement to allow at least one application"
+ " to start");
{code}
> RM recovery too slow due to LeafQueue#activateApplication()
> ---
>
> Key: YARN-5773
> URL: https://issues.apache.org/jira/browse/YARN-5773
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-5773.0001.patch, YARN-5773.0002.patch,
> YARN-5773.0004.patch, YARN-5773.0005.patch, YARN-5773.003.patch
>
>
> # Submit application 10K application to default queue.
> # All applications are in accepted state
> # Now restart resourcemanager
> For each application recovery {{LeafQueue#activateApplications()}} is
> invoked.Resulting in AM limit check to be done even before Node managers are
> getting registered.
> Total iteration for N application is about {{N(N+1)/2}} for {{10K}}
> application {{5000}} iterations causing time take for Rm to be active
> more than 10 min.
> Since NM resources are not yet added to during recovery we should skip
> {{activateApplicaiton()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5771) Provide option to send env to be whitelisted in ContainerLaunchContext
[
https://issues.apache.org/jira/browse/YARN-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612405#comment-15612405
]
Daniel Templeton commented on YARN-5771:
Sorry for restating the obvious. I missed the context of MAPREDUCE-6704.
I don't think that letting users control the whitelisting on a per-job basis is
the right answer. As far as I can tell, the problem we're trying to solve here
is that it should just work out of the box. The simplest and sanest answer,
IMO, is what you proposed on MAPREDUCE-6704: add {{HADOOP_MAPRED_HOME}} to
{{yarn.nodemanager.env-whitelist}} by default. I agree with [~aw] that users
aren't going to be offended by a little pollution if it means things just work.
> Provide option to send env to be whitelisted in ContainerLaunchContext
> ---
>
> Key: YARN-5771
> URL: https://issues.apache.org/jira/browse/YARN-5771
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: container-whitelist-env-wip.patch
>
>
> As per current implementation ENV to be white listed for container launch is
> are configured as part of {{yarn.nodemanager.env-whitelist}}
> Specific to container we cannot specify additional ENV properties to be
> whitelisted. As part of this jira we are providing an option to provide
> additional whitelist ENV.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5710) Fix inconsistent naming in class ResourceRequest
[ https://issues.apache.org/jira/browse/YARN-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612399#comment-15612399 ] Yufei Gu commented on YARN-5710: Thanks [~sunilg] and [~varun_saxena] for the review and commit. Thanks [~kasha] for the offline review. > Fix inconsistent naming in class ResourceRequest > > > Key: YARN-5710 > URL: https://issues.apache.org/jira/browse/YARN-5710 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Yufei Gu >Assignee: Yufei Gu >Priority: Trivial > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5710.001.patch, YARN-5710.002.patch > > > "node", "machine" and "host" are the same thing with different name in this > context. Consolidate them to "node". -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5773) RM recovery too slow due to LeafQueue#activateApplication()
[
https://issues.apache.org/jira/browse/YARN-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612297#comment-15612297
]
Bibin A Chundatt edited comment on YARN-5773 at 10/27/16 3:52 PM:
--
{quote}
When cluster resource is 0, this is unlikely to be a case.
{quote}
I did verify the same when cluster resource is empty and submit application.The
first application attempt is activated.
I think its should be fine keeping the same. Since the loop will run only
once.Hope in last patch no changes are required.
was (Author: bibinchundatt):
{quote}
When cluster resource is 0, this is unlikely to be a case.
{quote}
I did verify the same when cluster resource is empty and submit application.The
first application attempt is activated.
I think its should be fine keeping the same. Since the loop will run one once.
> RM recovery too slow due to LeafQueue#activateApplication()
> ---
>
> Key: YARN-5773
> URL: https://issues.apache.org/jira/browse/YARN-5773
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-5773.0001.patch, YARN-5773.0002.patch,
> YARN-5773.0004.patch, YARN-5773.0005.patch, YARN-5773.003.patch
>
>
> # Submit application 10K application to default queue.
> # All applications are in accepted state
> # Now restart resourcemanager
> For each application recovery {{LeafQueue#activateApplications()}} is
> invoked.Resulting in AM limit check to be done even before Node managers are
> getting registered.
> Total iteration for N application is about {{N(N+1)/2}} for {{10K}}
> application {{5000}} iterations causing time take for Rm to be active
> more than 10 min.
> Since NM resources are not yet added to during recovery we should skip
> {{activateApplicaiton()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5690) Integrate native services modules into maven build
[ https://issues.apache.org/jira/browse/YARN-5690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha updated YARN-5690: Fix Version/s: yarn-native-services > Integrate native services modules into maven build > -- > > Key: YARN-5690 > URL: https://issues.apache.org/jira/browse/YARN-5690 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Fix For: yarn-native-services > > Attachments: YARN-5690-yarn-native-services.001.patch, > YARN-5690-yarn-native-services.002.patch, > YARN-5690-yarn-native-services.003.patch > > > The yarn dist assembly should include jars for the new modules as well as > their new dependencies. We may want to create new lib directories in the > tarball for the dependencies of the slider-core and services API modules, to > avoid adding these dependencies into the general YARN classpath. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5773) RM recovery too slow due to LeafQueue#activateApplication()
[
https://issues.apache.org/jira/browse/YARN-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612297#comment-15612297
]
Bibin A Chundatt commented on YARN-5773:
{quote}
When cluster resource is 0, this is unlikely to be a case.
{quote}
I did verify the same when cluster resource is empty and submit application.The
first application attempt is activated.
I think its should be fine keeping the same. Since the loop will run one once.
> RM recovery too slow due to LeafQueue#activateApplication()
> ---
>
> Key: YARN-5773
> URL: https://issues.apache.org/jira/browse/YARN-5773
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-5773.0001.patch, YARN-5773.0002.patch,
> YARN-5773.0004.patch, YARN-5773.0005.patch, YARN-5773.003.patch
>
>
> # Submit application 10K application to default queue.
> # All applications are in accepted state
> # Now restart resourcemanager
> For each application recovery {{LeafQueue#activateApplications()}} is
> invoked.Resulting in AM limit check to be done even before Node managers are
> getting registered.
> Total iteration for N application is about {{N(N+1)/2}} for {{10K}}
> application {{5000}} iterations causing time take for Rm to be active
> more than 10 min.
> Since NM resources are not yet added to during recovery we should skip
> {{activateApplicaiton()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5490) [YARN-3368] Fix various alignment issues and broken breadcrumb link in Node page
[ https://issues.apache.org/jira/browse/YARN-5490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-5490: -- Summary: [YARN-3368] Fix various alignment issues and broken breadcrumb link in Node page (was: [YARN-3368] Fix various alignment issues in Node page and the broken breadcrumb link) > [YARN-3368] Fix various alignment issues and broken breadcrumb link in Node > page > - > > Key: YARN-5490 > URL: https://issues.apache.org/jira/browse/YARN-5490 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Akhil PB > Attachments: YARN-5490-YARN-3368.001.patch > > > Few alignment issue in nodes page > breadcrumb view is not showing the node table when clicked on Nodes option. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5785) [YARN-3368] Accessing applications and containers list from Node page is throwing few exceptions in console
[ https://issues.apache.org/jira/browse/YARN-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612250#comment-15612250 ] Sunil G commented on YARN-5785: --- Thanks [~akhilpb] for the patch. Looks fine for me. This is a change in adapter and serializer. So I ll do some more tests before commit. [~Sreenath], could you please help to review this change. > [YARN-3368] Accessing applications and containers list from Node page is > throwing few exceptions in console > --- > > Key: YARN-5785 > URL: https://issues.apache.org/jira/browse/YARN-5785 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Sunil G >Assignee: Akhil PB > Attachments: YARN-5785-YARN-3368.001.patch > > > In node page, "List of Applications" and "List of Containers" links are > causing few error logs in console. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5490) [YARN-3368] Fix various alignment issues in Node page and the broken breadcrumb link
[ https://issues.apache.org/jira/browse/YARN-5490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612223#comment-15612223 ] Sunil G commented on YARN-5490: --- +1. Tested in local cluster. Node page breadcrumbs are working fine. Committing the same. > [YARN-3368] Fix various alignment issues in Node page and the broken > breadcrumb link > > > Key: YARN-5490 > URL: https://issues.apache.org/jira/browse/YARN-5490 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Akhil PB > Attachments: YARN-5490-YARN-3368.001.patch > > > Few alignment issue in nodes page > breadcrumb view is not showing the node table when clicked on Nodes option. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5697) Use CliParser to parse options in RMAdminCLI
[ https://issues.apache.org/jira/browse/YARN-5697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612195#comment-15612195 ] Sunil G commented on YARN-5697: --- Yes. I also feel that if any sub options params has some value to be read, then it should immediately follow the command. This will help the readability. I think i prefer the approach suggested above. > Use CliParser to parse options in RMAdminCLI > > > Key: YARN-5697 > URL: https://issues.apache.org/jira/browse/YARN-5697 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Tao Jie >Assignee: Tao Jie > Attachments: YARN-5697.001.patch, YARN-5697.002.patch, > YARN-5697.003.patch > > > As discussed in YARN-4855, it is better to use CliParser rather than args to > parse command line options in RMAdminCli. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5375) invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures
[ https://issues.apache.org/jira/browse/YARN-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612189#comment-15612189 ] sandflee commented on YARN-5375: Thanks [~rohithsharma], yes, MockRM#createSchedulerEventDispatcher triggers the deadlock. NodeAddEvent used to left in schedule event queue now get processed. agree that using resourcemanager.handle(nodeUpdate) could solve this bug. and this test seems very special, only rm dispatcher started. > invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures > -- > > Key: YARN-5375 > URL: https://issues.apache.org/jira/browse/YARN-5375 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: sandflee >Assignee: sandflee > Attachments: YARN-5375.01.patch, YARN-5375.03.patch, > YARN-5375.04.patch, YARN-5375.05.patch, YARN-5375.06.patch, > YARN-5375.07-drain-statestore.patch, YARN-5375.07-sync-statestore.patch, > YARN-5375.08.patch > > > seen many test failures related to RMApp/RMAppattempt comes to some state but > some event are not processed in rm event queue or scheduler event queue, > cause test failure, seems we could implicitly invokes drainEvents(should also > drain sheduler event) in some mockRM method like waitForState -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2009) Priority support for preemption in ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612098#comment-15612098 ] Sunil G commented on YARN-2009: --- Thanks [~eepayne]. Yes. I am working on scheduler's user-limit improvement (jira will be raised separately) and in YARN-2113. > Priority support for preemption in ProportionalCapacityPreemptionPolicy > --- > > Key: YARN-2009 > URL: https://issues.apache.org/jira/browse/YARN-2009 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Devaraj K >Assignee: Sunil G > Attachments: YARN-2009.0001.patch, YARN-2009.0002.patch, > YARN-2009.0003.patch, YARN-2009.0004.patch, YARN-2009.0005.patch, > YARN-2009.0006.patch, YARN-2009.0007.patch, YARN-2009.0008.patch, > YARN-2009.0009.patch, YARN-2009.0010.patch, YARN-2009.0011.patch, > YARN-2009.0012.patch, YARN-2009.0013.patch, YARN-2009.0014.patch, > YARN-2009.0015.patch > > > While preempting containers based on the queue ideal assignment, we may need > to consider preempting the low priority application containers first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-1468) TestRMRestart.testRMRestartWaitForPreviousAMToFinish get failed.
[
https://issues.apache.org/jira/browse/YARN-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe resolved YARN-1468.
--
Resolution: Duplicate
Closing this as a duplicate of YARN-5416 since that other JIRA has a proposed
patch and recent discussion.
> TestRMRestart.testRMRestartWaitForPreviousAMToFinish get failed.
>
>
> Key: YARN-1468
> URL: https://issues.apache.org/jira/browse/YARN-1468
> Project: Hadoop YARN
> Issue Type: Test
> Components: resourcemanager
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Critical
>
> Log is as following:
> {code}
> Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 149.968 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testRMRestartWaitForPreviousAMToFinish(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
> Time elapsed: 44.197 sec <<< FAILURE!
> junit.framework.AssertionFailedError: AppAttempt state is not correct
> (timedout) expected: but was:
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:292)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.launchAM(TestRMRestart.java:826)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartWaitForPreviousAMToFinish(TestRMRestart.java:464)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3933) Race condition when calling AbstractYarnScheduler.completedContainer.
[ https://issues.apache.org/jira/browse/YARN-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612089#comment-15612089 ] Sunil G commented on YARN-3933: --- Test case failure is tracked in YARN-5043 > Race condition when calling AbstractYarnScheduler.completedContainer. > - > > Key: YARN-3933 > URL: https://issues.apache.org/jira/browse/YARN-3933 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.0, 2.7.0, 2.5.2, 2.7.1 >Reporter: Lavkesh Lahngir >Assignee: Shiwei Guo > Attachments: YARN-3933.001.patch, YARN-3933.002.patch, > YARN-3933.003.patch, YARN-3933.004.patch, YARN-3933.005.patch > > > In our cluster we are seeing available memory and cores being negative. > Initial inspection: > Scenario no. 1: > In capacity scheduler the method allocateContainersToNode() checks if > there are excess reservation of containers for an application, and they are > no longer needed then it calls queue.completedContainer() which causes > resources being negative. And they were never assigned in the first place. > I am still looking through the code. Can somebody suggest how to simulate > excess containers assignments ? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-3933) Race condition when calling AbstractYarnScheduler.completedContainer.
[ https://issues.apache.org/jira/browse/YARN-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612089#comment-15612089 ] Sunil G edited comment on YARN-3933 at 10/27/16 2:43 PM: - Test case failure is tracked in YARN-5043. [~kasha] pls help to review the same. was (Author: sunilg): Test case failure is tracked in YARN-5043 > Race condition when calling AbstractYarnScheduler.completedContainer. > - > > Key: YARN-3933 > URL: https://issues.apache.org/jira/browse/YARN-3933 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.0, 2.7.0, 2.5.2, 2.7.1 >Reporter: Lavkesh Lahngir >Assignee: Shiwei Guo > Attachments: YARN-3933.001.patch, YARN-3933.002.patch, > YARN-3933.003.patch, YARN-3933.004.patch, YARN-3933.005.patch > > > In our cluster we are seeing available memory and cores being negative. > Initial inspection: > Scenario no. 1: > In capacity scheduler the method allocateContainersToNode() checks if > there are excess reservation of containers for an application, and they are > no longer needed then it calls queue.completedContainer() which causes > resources being negative. And they were never assigned in the first place. > I am still looking through the code. Can somebody suggest how to simulate > excess containers assignments ? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5767) Fix the order that resources are cleaned up from the local Public/Private caches
[
https://issues.apache.org/jira/browse/YARN-5767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612061#comment-15612061
]
Jason Lowe commented on YARN-5767:
--
+1 lgtm. I'll commit this tomorrow if there are no objections.
> Fix the order that resources are cleaned up from the local Public/Private
> caches
>
>
> Key: YARN-5767
> URL: https://issues.apache.org/jira/browse/YARN-5767
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.0, 3.0.0-alpha1
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Attachments: YARN-5767-trunk-v1.patch, YARN-5767-trunk-v2.patch,
> YARN-5767-trunk-v3.patch, YARN-5767-trunk-v4.patch
>
>
> If you look at {{ResourceLocalizationService#handleCacheCleanup}}, you can
> see that public resources are added to the {{ResourceRetentionSet}} first
> followed by private resources:
> {code:java}
> private void handleCacheCleanup(LocalizationEvent event) {
> ResourceRetentionSet retain =
> new ResourceRetentionSet(delService, cacheTargetSize);
> retain.addResources(publicRsrc);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Resource cleanup (public) " + retain);
> }
> for (LocalResourcesTracker t : privateRsrc.values()) {
> retain.addResources(t);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Resource cleanup " + t.getUser() + ":" + retain);
> }
> }
> //TODO Check if appRsrcs should also be added to the retention set.
> }
> {code}
> Unfortunately, if we look at {{ResourceRetentionSet#addResources}} we see
> that this means public resources are deleted first until the target cache
> size is met:
> {code:java}
> public void addResources(LocalResourcesTracker newTracker) {
> for (LocalizedResource resource : newTracker) {
> currentSize += resource.getSize();
> if (resource.getRefCount() > 0) {
> // always retain resources in use
> continue;
> }
> retain.put(resource, newTracker);
> }
> for (Iterator> i =
> retain.entrySet().iterator();
>currentSize - delSize > targetSize && i.hasNext();) {
> Map.Entry rsrc = i.next();
> LocalizedResource resource = rsrc.getKey();
> LocalResourcesTracker tracker = rsrc.getValue();
> if (tracker.remove(resource, delService)) {
> delSize += resource.getSize();
> i.remove();
> }
> }
> }
> {code}
> The result of this is that resources in the private cache are only deleted in
> the cases where:
> # The cache size is larger than the target cache size and the public cache is
> empty.
> # The cache size is larger than the target cache size and everything in the
> public cache is being used by a running container.
> For clusters that primarily use the public cache (i.e. make use of the shared
> cache), this means that the most commonly used resources can be deleted
> before old resources in the private cache. Furthermore, the private cache can
> continue to grow over time causing more and more churn in the public cache.
> Additionally, the same problem exists within the private cache. Since
> resources are added to the retention set on a user by user basis, resources
> will get cleaned up one user at a time in the order that privateRsrc.values()
> returns the LocalResourcesTracker. So if user1 has 10MB in their cache and
> user2 has 100MB in their cache and the target size of the cache is 50MB,
> user1 could potentially have their entire cache removed before anything is
> deleted from the user2 cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3933) Race condition when calling AbstractYarnScheduler.completedContainer.
[
https://issues.apache.org/jira/browse/YARN-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612034#comment-15612034
]
Hadoop QA commented on YARN-3933:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 20s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 5 new + 274 unchanged - 2 fixed = 279 total (was 276)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 7s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-3933 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835574/YARN-3933.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e732b5fc242e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b439517 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/13555/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/13555/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/13555/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13555/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-y
[jira] [Commented] (YARN-2009) Priority support for preemption in ProportionalCapacityPreemptionPolicy
[
https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611992#comment-15611992
]
Eric Payne commented on YARN-2009:
--
{quote}
My suggestion is： commit existing patch, at least it works for single user and
(I think) it doesn't have fundamental issue. We can fix it in the next patch
with highest priority. Does this sound OK
{quote}
Yes, that's fine. I assume our next step is to begin working on YARN-2113?
> Priority support for preemption in ProportionalCapacityPreemptionPolicy
> ---
>
> Key: YARN-2009
> URL: https://issues.apache.org/jira/browse/YARN-2009
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Devaraj K
>Assignee: Sunil G
> Attachments: YARN-2009.0001.patch, YARN-2009.0002.patch,
> YARN-2009.0003.patch, YARN-2009.0004.patch, YARN-2009.0005.patch,
> YARN-2009.0006.patch, YARN-2009.0007.patch, YARN-2009.0008.patch,
> YARN-2009.0009.patch, YARN-2009.0010.patch, YARN-2009.0011.patch,
> YARN-2009.0012.patch, YARN-2009.0013.patch, YARN-2009.0014.patch,
> YARN-2009.0015.patch
>
>
> While preempting containers based on the queue ideal assignment, we may need
> to consider preempting the low priority application containers first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5420) Delete org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as its not necessary
[ https://issues.apache.org/jira/browse/YARN-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611860#comment-15611860 ] Hudson commented on YARN-5420: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10704 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10704/]) YARN-5420. Delete (naganarasimha_gr: rev b3c15e4ef763ebc4b033c686114fe627350824ac) * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Priority.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerHealth.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java > Delete org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as its > not necessary > --- > > Key: YARN-5420 > URL: https://issues.apache.org/jira/browse/YARN-5420 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5420.001.patch, YARN-5420.002.patch, > YARN-5420.003.patch > > > org.apache.hadoop.yarn.server.resourcemanager.resource.Priority is no longer > used after YARN-5392. It can be deprecated or removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5787) Deadlock cycle in FairScheduler
[
https://issues.apache.org/jira/browse/YARN-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611848#comment-15611848
]
Rohith Sharma K S commented on YARN-5787:
-
Below is the dead lock trace
{noformat}
"AsyncDispatcher event handler":
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.recoverContainersOnNode(AbstractYarnScheduler.java:390)
- waiting to lock <0x00076f5923c8> (a
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.addNode(FairScheduler.java:917)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1223)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM$8.handle(MockRM.java:1100)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM$8.handle(MockRM.java:1)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:187)
at
org.apache.hadoop.yarn.event.DrainDispatcher$1.run(DrainDispatcher.java:75)
at java.lang.Thread.run(Thread.java:745)
"main":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00076f6d7148> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:1044)
- locked <0x00076f5923c8> (a
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1238)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testResourceUpdateDecommissioningNode(TestFairScheduler.java:4741)
{noformat}
> Deadlock cycle in FairScheduler
>
>
> Key: YARN-5787
> URL: https://issues.apache.org/jira/browse/YARN-5787
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Rohith Sharma K S
>
> While testing YARN-5375 patch, found a dead lock cycle in fairscheduler.
> Basically it was cause by patch. cc:/[~sandflee]
> Even though in reality this scenario does not happen, it would be better to
> handle it from fairscheduler. I am not sure whether it was noticed earlier,
> but just thought it should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5787) Deadlock cycle in FairScheduler
Rohith Sharma K S created YARN-5787: --- Summary: Deadlock cycle in FairScheduler Key: YARN-5787 URL: https://issues.apache.org/jira/browse/YARN-5787 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Reporter: Rohith Sharma K S While testing YARN-5375 patch, found a dead lock cycle in fairscheduler. Basically it was cause by patch. cc:/[~sandflee] Even though in reality this scenario does not happen, it would be better to handle it from fairscheduler. I am not sure whether it was noticed earlier, but just thought it should be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3477) TimelineClientImpl swallows exceptions
[ https://issues.apache.org/jira/browse/YARN-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611832#comment-15611832 ] Varun Saxena commented on YARN-3477: There is an open PR which Jenkins seems to be picking up. [~ste...@apache.org], can you rebase there or close the PR ? > TimelineClientImpl swallows exceptions > -- > > Key: YARN-3477 > URL: https://issues.apache.org/jira/browse/YARN-3477 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0, 2.7.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: YARN-3477-001.patch, YARN-3477-002.patch, > YARN-3477-trunk.003.patch, YARN-3477-trunk.004.patch > > > If timeline client fails more than the retry count, the original exception is > not thrown. Instead some runtime exception is raised saying "retries run out" > # the failing exception should be rethrown, ideally via > NetUtils.wrapException to include URL of the failing endpoing > # Otherwise, the raised RTE should (a) state that URL and (b) set the > original fault as the inner cause -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3933) Race condition when calling AbstractYarnScheduler.completedContainer.
[ https://issues.apache.org/jira/browse/YARN-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-3933: -- Attachment: YARN-3933.005.patch rebased to latest trunk. pls review. > Race condition when calling AbstractYarnScheduler.completedContainer. > - > > Key: YARN-3933 > URL: https://issues.apache.org/jira/browse/YARN-3933 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.0, 2.7.0, 2.5.2, 2.7.1 >Reporter: Lavkesh Lahngir >Assignee: Shiwei Guo > Attachments: YARN-3933.001.patch, YARN-3933.002.patch, > YARN-3933.003.patch, YARN-3933.004.patch, YARN-3933.005.patch > > > In our cluster we are seeing available memory and cores being negative. > Initial inspection: > Scenario no. 1: > In capacity scheduler the method allocateContainersToNode() checks if > there are excess reservation of containers for an application, and they are > no longer needed then it calls queue.completedContainer() which causes > resources being negative. And they were never assigned in the first place. > I am still looking through the code. Can somebody suggest how to simulate > excess containers assignments ? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5705) [YARN-3368] Add support for Timeline V2 to new web UI
[
https://issues.apache.org/jira/browse/YARN-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611800#comment-15611800
]
Hadoop QA commented on YARN-5705:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} YARN-5705 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5705 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835573/YARN-5705.012.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13554/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [YARN-3368] Add support for Timeline V2 to new web UI
> -
>
> Key: YARN-5705
> URL: https://issues.apache.org/jira/browse/YARN-5705
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Sunil G
>Assignee: Akhil PB
> Attachments: YARN-5705.001.patch, YARN-5705.002.patch,
> YARN-5705.003.patch, YARN-5705.004.patch, YARN-5705.005.patch,
> YARN-5705.006.patch, YARN-5705.007.patch, YARN-5705.008.patch,
> YARN-5705.009.patch, YARN-5705.010.patch, YARN-5705.011.patch,
> YARN-5705.012.patch
>
>
> Integrate timeline v2 to YARN-3368. This is a clone JIRA for YARN-4097
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5420) Delete org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as its not necessary
[ https://issues.apache.org/jira/browse/YARN-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611797#comment-15611797 ] Naganarasimha G R commented on YARN-5420: - Thanks for the contribution [~sunilg], committed the patch to trunk and branch-2 > Delete org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as its > not necessary > --- > > Key: YARN-5420 > URL: https://issues.apache.org/jira/browse/YARN-5420 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G >Priority: Minor > Attachments: YARN-5420.001.patch, YARN-5420.002.patch, > YARN-5420.003.patch > > > org.apache.hadoop.yarn.server.resourcemanager.resource.Priority is no longer > used after YARN-5392. It can be deprecated or removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5705) [YARN-3368] Add support for Timeline V2 to new web UI
[ https://issues.apache.org/jira/browse/YARN-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB updated YARN-5705: --- Attachment: YARN-5705.012.patch > [YARN-3368] Add support for Timeline V2 to new web UI > - > > Key: YARN-5705 > URL: https://issues.apache.org/jira/browse/YARN-5705 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Akhil PB > Attachments: YARN-5705.001.patch, YARN-5705.002.patch, > YARN-5705.003.patch, YARN-5705.004.patch, YARN-5705.005.patch, > YARN-5705.006.patch, YARN-5705.007.patch, YARN-5705.008.patch, > YARN-5705.009.patch, YARN-5705.010.patch, YARN-5705.011.patch, > YARN-5705.012.patch > > > Integrate timeline v2 to YARN-3368. This is a clone JIRA for YARN-4097 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5375) invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures
[
https://issues.apache.org/jira/browse/YARN-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611778#comment-15611778
]
Rohith Sharma K S commented on YARN-5375:
-
I debugged more finding reason for deadlock. It is because, patch adds
MockRM#createSchedulerEventDispatcher. I believe sleep might not solve
deadlock.
One of the solution I think is since after this patch however scheduler event
processing becomes sequential, need to change the test code to trigger an event
rather than {{resourceManager.getResourceScheduler().handle(nodeUpdate);}}
> invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures
> --
>
> Key: YARN-5375
> URL: https://issues.apache.org/jira/browse/YARN-5375
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: sandflee
>Assignee: sandflee
> Attachments: YARN-5375.01.patch, YARN-5375.03.patch,
> YARN-5375.04.patch, YARN-5375.05.patch, YARN-5375.06.patch,
> YARN-5375.07-drain-statestore.patch, YARN-5375.07-sync-statestore.patch,
> YARN-5375.08.patch
>
>
> seen many test failures related to RMApp/RMAppattempt comes to some state but
> some event are not processed in rm event queue or scheduler event queue,
> cause test failure, seems we could implicitly invokes drainEvents(should also
> drain sheduler event) in some mockRM method like waitForState
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5420) Delete org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as its not necessary
[ https://issues.apache.org/jira/browse/YARN-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-5420: Summary: Delete org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as its not necessary (was: Deprecate org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as SchedulerKey wrapper is introduced) > Delete org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as its > not necessary > --- > > Key: YARN-5420 > URL: https://issues.apache.org/jira/browse/YARN-5420 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G >Priority: Minor > Attachments: YARN-5420.001.patch, YARN-5420.002.patch, > YARN-5420.003.patch > > > org.apache.hadoop.yarn.server.resourcemanager.resource.Priority is no longer > used after YARN-5392. It can be deprecated or removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5420) Deprecate org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as SchedulerKey wrapper is introduced
[
https://issues.apache.org/jira/browse/YARN-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611760#comment-15611760
]
Hadoop QA commented on YARN-5420:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 19 new + 747 unchanged - 30 fixed = 766 total (was 777)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m
17s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5420 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835559/YARN-5420.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7d471ede6525 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6c88309 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/13551/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13551/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13551/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Deprecate org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as
> SchedulerKey wrapper is introduced
> --
[jira] [Assigned] (YARN-5494) Nodes page throws "Sorry Error Occurred" message
[
https://issues.apache.org/jira/browse/YARN-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akhil PB reassigned YARN-5494:
--
Assignee: Akhil PB
> Nodes page throws "Sorry Error Occurred" message
>
>
> Key: YARN-5494
> URL: https://issues.apache.org/jira/browse/YARN-5494
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Yesha Vora
>Assignee: Akhil PB
>Priority: Critical
>
> Steps to reproduce:
> * Click on Nodes. This page will list nodes of the cluster
> * Click on one the nodes such as node1. ( It will redirect to
> http://:4200/#/yarn-node/:31924/:8042 url)
> This url prompts "Sorry Error Occurred" error.
> {code}jquery.js:8630 XMLHttpRequest cannot load
> http://xxx:xxx:8042/ws/v1/node. Cross origin requests are only supported for
> protocol schemes: http, data, chrome, chrome-extension, https,
> chrome-extension-resource.send @ jquery.js:8630
> ember.debug.js:30877 Error: Adapter operation failed
> at new Error (native)
> at Error.EmberError (http://xxx:4200/assets/vendor.js:25278:21)
> at Error.ember$data$lib$adapters$errors$$AdapterError
> (http://xxx:4200/assets/vendor.js:91198:50)
> at Class.handleResponse (http://xxx:4200/assets/vendor.js:92494:16)
> at Class.hash.error (http://xxx:4200/assets/vendor.js:92574:33)
> at fire (http://xxx:4200/assets/vendor.js:3306:30)
> at Object.fireWith [as rejectWith]
> (http://xxx:4200/assets/vendor.js:3418:7)
> at done (http://xxx:4200/assets/vendor.js:8473:14)
> at XMLHttpRequest. (http://xxx:4200/assets/vendor.js:8806:9)
> at Object.send (http://xxx:4200/assets/vendor.js:8837:10)onerrorDefault @
> ember.debug.js:30877{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5785) [YARN-3368] Accessing applications and containers list from Node page is throwing few exceptions in console
[
https://issues.apache.org/jira/browse/YARN-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611736#comment-15611736
]
Hadoop QA commented on YARN-5785:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:5a4801a |
| JIRA Issue | YARN-5785 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835565/YARN-5785-YARN-3368.001.patch
|
| Optional Tests | asflicense |
| uname | Linux 2e557022e119 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3368 / a10cf98 |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13553/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [YARN-3368] Accessing applications and containers list from Node page is
> throwing few exceptions in console
> ---
>
> Key: YARN-5785
> URL: https://issues.apache.org/jira/browse/YARN-5785
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Sunil G
>Assignee: Akhil PB
> Attachments: YARN-5785-YARN-3368.001.patch
>
>
> In node page, "List of Applications" and "List of Containers" links are
> causing few error logs in console.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5785) [YARN-3368] Accessing applications and containers list from Node page is throwing few exceptions in console
[ https://issues.apache.org/jira/browse/YARN-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB updated YARN-5785: --- Attachment: YARN-5785-YARN-3368.001.patch > [YARN-3368] Accessing applications and containers list from Node page is > throwing few exceptions in console > --- > > Key: YARN-5785 > URL: https://issues.apache.org/jira/browse/YARN-5785 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Sunil G >Assignee: Akhil PB > Attachments: YARN-5785-YARN-3368.001.patch > > > In node page, "List of Applications" and "List of Containers" links are > causing few error logs in console. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3477) TimelineClientImpl swallows exceptions
[
https://issues.apache.org/jira/browse/YARN-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611711#comment-15611711
]
Hadoop QA commented on YARN-3477:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red} YARN-3477 does not apply to branch-2. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-3477 |
| GITHUB PR | https://github.com/apache/hadoop/pull/47 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13552/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TimelineClientImpl swallows exceptions
> --
>
> Key: YARN-3477
> URL: https://issues.apache.org/jira/browse/YARN-3477
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: YARN-3477-001.patch, YARN-3477-002.patch,
> YARN-3477-trunk.003.patch, YARN-3477-trunk.004.patch
>
>
> If timeline client fails more than the retry count, the original exception is
> not thrown. Instead some runtime exception is raised saying "retries run out"
> # the failing exception should be rethrown, ideally via
> NetUtils.wrapException to include URL of the failing endpoing
> # Otherwise, the raised RTE should (a) state that URL and (b) set the
> original fault as the inner cause
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3477) TimelineClientImpl swallows exceptions
[ https://issues.apache.org/jira/browse/YARN-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3477: --- Attachment: YARN-3477-trunk.004.patch > TimelineClientImpl swallows exceptions > -- > > Key: YARN-3477 > URL: https://issues.apache.org/jira/browse/YARN-3477 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0, 2.7.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: YARN-3477-001.patch, YARN-3477-002.patch, > YARN-3477-trunk.003.patch, YARN-3477-trunk.004.patch > > > If timeline client fails more than the retry count, the original exception is > not thrown. Instead some runtime exception is raised saying "retries run out" > # the failing exception should be rethrown, ideally via > NetUtils.wrapException to include URL of the failing endpoing > # Otherwise, the raised RTE should (a) state that URL and (b) set the > original fault as the inner cause -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3477) TimelineClientImpl swallows exceptions
[ https://issues.apache.org/jira/browse/YARN-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611700#comment-15611700 ] Varun Saxena commented on YARN-3477: Rebasing the patch. Will look into it. > TimelineClientImpl swallows exceptions > -- > > Key: YARN-3477 > URL: https://issues.apache.org/jira/browse/YARN-3477 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.6.0, 2.7.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: YARN-3477-001.patch, YARN-3477-002.patch, > YARN-3477-trunk.003.patch, YARN-3477-trunk.004.patch > > > If timeline client fails more than the retry count, the original exception is > not thrown. Instead some runtime exception is raised saying "retries run out" > # the failing exception should be rethrown, ideally via > NetUtils.wrapException to include URL of the failing endpoing > # Otherwise, the raised RTE should (a) state that URL and (b) set the > original fault as the inner cause -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5786) RM REST API documentation does not capture labels and scheduler activities related endpoints
[ https://issues.apache.org/jira/browse/YARN-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611665#comment-15611665 ] Sunil G commented on YARN-5786: --- Yes. There is an ongoing jira for that. YARN-5256 > RM REST API documentation does not capture labels and scheduler activities > related endpoints > - > > Key: YARN-5786 > URL: https://issues.apache.org/jira/browse/YARN-5786 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Saxena >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5773) RM recovery too slow due to LeafQueue#activateApplication()
[
https://issues.apache.org/jira/browse/YARN-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611663#comment-15611663
]
Sunil G commented on YARN-5773:
---
I also think {{getNumActiveApplications}} is not needed to be checked during
Recovery call flow.
> RM recovery too slow due to LeafQueue#activateApplication()
> ---
>
> Key: YARN-5773
> URL: https://issues.apache.org/jira/browse/YARN-5773
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-5773.0001.patch, YARN-5773.0002.patch,
> YARN-5773.0004.patch, YARN-5773.0005.patch, YARN-5773.003.patch
>
>
> # Submit application 10K application to default queue.
> # All applications are in accepted state
> # Now restart resourcemanager
> For each application recovery {{LeafQueue#activateApplications()}} is
> invoked.Resulting in AM limit check to be done even before Node managers are
> getting registered.
> Total iteration for N application is about {{N(N+1)/2}} for {{10K}}
> application {{5000}} iterations causing time take for Rm to be active
> more than 10 min.
> Since NM resources are not yet added to during recovery we should skip
> {{activateApplicaiton()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5420) Deprecate org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as SchedulerKey wrapper is introduced
[ https://issues.apache.org/jira/browse/YARN-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611659#comment-15611659 ] Naganarasimha G R commented on YARN-5420: - +1, Latest patch LGTM, waiting for the pending jenkins will commit it once its fine ! > Deprecate org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as > SchedulerKey wrapper is introduced > --- > > Key: YARN-5420 > URL: https://issues.apache.org/jira/browse/YARN-5420 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G >Priority: Minor > Attachments: YARN-5420.001.patch, YARN-5420.002.patch, > YARN-5420.003.patch > > > org.apache.hadoop.yarn.server.resourcemanager.resource.Priority is no longer > used after YARN-5392. It can be deprecated or removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5773) RM recovery too slow due to LeafQueue#activateApplication()
[
https://issues.apache.org/jira/browse/YARN-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611658#comment-15611658
]
Varun Saxena commented on YARN-5773:
bq. Earlier implementation also followed the same. if i understand correctly
Earlier implementation was to handle the case where AM resource limit has been
set too low. When cluster resource is 0, this is unlikely to be a case.
> RM recovery too slow due to LeafQueue#activateApplication()
> ---
>
> Key: YARN-5773
> URL: https://issues.apache.org/jira/browse/YARN-5773
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-5773.0001.patch, YARN-5773.0002.patch,
> YARN-5773.0004.patch, YARN-5773.0005.patch, YARN-5773.003.patch
>
>
> # Submit application 10K application to default queue.
> # All applications are in accepted state
> # Now restart resourcemanager
> For each application recovery {{LeafQueue#activateApplications()}} is
> invoked.Resulting in AM limit check to be done even before Node managers are
> getting registered.
> Total iteration for N application is about {{N(N+1)/2}} for {{10K}}
> application {{5000}} iterations causing time take for Rm to be active
> more than 10 min.
> Since NM resources are not yet added to during recovery we should skip
> {{activateApplicaiton()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5757) RM REST API documentation is not up to date
[ https://issues.apache.org/jira/browse/YARN-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611656#comment-15611656 ] Hudson commented on YARN-5757: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #10702 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10702/]) YARN-5757. RM REST API documentation is not up to date (Miklos Szegedi (varunsaxena: rev 8bd6d77a416dd15172f4539cb9249203507172c1) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerRest.md * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/site/markdown/HistoryServerRest.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md > RM REST API documentation is not up to date > --- > > Key: YARN-5757 > URL: https://issues.apache.org/jira/browse/YARN-5757 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, yarn >Affects Versions: 2.7.3, 3.0.0-alpha1 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Trivial > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5757.000.patch, YARN-5757.001.patch > > > For an example please refer to this field that does not exist since YARN-686: > healthStatus string The health status of the node - Healthy or Unhealthy -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3848) TestNodeLabelContainerAllocation is not timing out
[ https://issues.apache.org/jira/browse/YARN-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611653#comment-15611653 ] Hudson commented on YARN-3848: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #10702 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10702/]) YARN-3848. TestNodeLabelContainerAllocation is timing out. Contributed (naganarasimha_gr: rev 6c8830992c734313fc95ac9988539c4d813c3581) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java > TestNodeLabelContainerAllocation is not timing out > -- > > Key: YARN-3848 > URL: https://issues.apache.org/jira/browse/YARN-3848 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Reporter: Jason Lowe >Assignee: Varun Saxena > Attachments: YARN-3848.01.patch, YARN-3848.02.patch, test_output.txt > > > A number of builds, pre-commit and otherwise, have been failing recently > because TestNodeLabelContainerAllocation has timed out. See > https://builds.apache.org/job/Hadoop-Yarn-trunk/969/, YARN-3830, YARN-3802, > or YARN-3826 for examples. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5757) RM REST API documentation is not up to date
[ https://issues.apache.org/jira/browse/YARN-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611651#comment-15611651 ] Varun Saxena commented on YARN-5757: Committed to trunk, branch-2. Thanks Miklos Szegedi for your contribution. > RM REST API documentation is not up to date > --- > > Key: YARN-5757 > URL: https://issues.apache.org/jira/browse/YARN-5757 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, yarn >Affects Versions: 2.7.3, 3.0.0-alpha1 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Trivial > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5757.000.patch, YARN-5757.001.patch > > > For an example please refer to this field that does not exist since YARN-686: > healthStatus string The health status of the node - Healthy or Unhealthy -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3848) TestNodeLabelContainerAllocation is not timing out
[ https://issues.apache.org/jira/browse/YARN-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-3848: Summary: TestNodeLabelContainerAllocation is not timing out (was: TestNodeLabelContainerAllocation is timing out) > TestNodeLabelContainerAllocation is not timing out > -- > > Key: YARN-3848 > URL: https://issues.apache.org/jira/browse/YARN-3848 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Reporter: Jason Lowe >Assignee: Varun Saxena > Attachments: YARN-3848.01.patch, YARN-3848.02.patch, test_output.txt > > > A number of builds, pre-commit and otherwise, have been failing recently > because TestNodeLabelContainerAllocation has timed out. See > https://builds.apache.org/job/Hadoop-Yarn-trunk/969/, YARN-3830, YARN-3802, > or YARN-3826 for examples. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5757) RM REST API documentation is not up to date
[ https://issues.apache.org/jira/browse/YARN-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611648#comment-15611648 ] Varun Saxena commented on YARN-5757: Raised YARN-5786 > RM REST API documentation is not up to date > --- > > Key: YARN-5757 > URL: https://issues.apache.org/jira/browse/YARN-5757 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, yarn >Affects Versions: 2.7.3, 3.0.0-alpha1 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Trivial > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5757.000.patch, YARN-5757.001.patch > > > For an example please refer to this field that does not exist since YARN-686: > healthStatus string The health status of the node - Healthy or Unhealthy -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3848) TestNodeLabelContainerAllocation is timing out
[ https://issues.apache.org/jira/browse/YARN-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611646#comment-15611646 ] Naganarasimha G R commented on YARN-3848: - Thanks for the patch [~varun_saxena] and review from [~wangda], i have committed to trunk, 2.8 and 2.9. > TestNodeLabelContainerAllocation is timing out > -- > > Key: YARN-3848 > URL: https://issues.apache.org/jira/browse/YARN-3848 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Reporter: Jason Lowe >Assignee: Varun Saxena > Attachments: YARN-3848.01.patch, YARN-3848.02.patch, test_output.txt > > > A number of builds, pre-commit and otherwise, have been failing recently > because TestNodeLabelContainerAllocation has timed out. See > https://builds.apache.org/job/Hadoop-Yarn-trunk/969/, YARN-3830, YARN-3802, > or YARN-3826 for examples. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5786) RM REST API documentation does not capture labels and scheduler activities related endpoints
Varun Saxena created YARN-5786: -- Summary: RM REST API documentation does not capture labels and scheduler activities related endpoints Key: YARN-5786 URL: https://issues.apache.org/jira/browse/YARN-5786 Project: Hadoop YARN Issue Type: Bug Reporter: Varun Saxena Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5786) RM REST API documentation does not capture labels and scheduler activities related endpoints
[ https://issues.apache.org/jira/browse/YARN-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611628#comment-15611628 ] Varun Saxena commented on YARN-5786: [~sunilg], IIUC you are updating some node label REST endpoints. Maybe this can be handled after that. > RM REST API documentation does not capture labels and scheduler activities > related endpoints > - > > Key: YARN-5786 > URL: https://issues.apache.org/jira/browse/YARN-5786 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Saxena >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
