[jira] [Commented] (YARN-6816) Add functionality to remove labels from nodes
[ https://issues.apache.org/jira/browse/YARN-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088759#comment-16088759 ] Jonathan Hung commented on YARN-6816: - I see, thanks. I suppose this will be more relevant when multiple labels per node is supported. > Add functionality to remove labels from nodes > - > > Key: YARN-6816 > URL: https://issues.apache.org/jira/browse/YARN-6816 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung > > Seems in YARN-2505 functionality to remove labels from nodes was missed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5892) Support user-specific minimum user limit percentage in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-5892:
-
Attachment: YARN-5892.branch-2.8.016.patch
I cherry-picked the trunk commit to branch-2 for this JIRA.
I am now attaching YARN-5892.branch-2.8.016.patch, which had to be refactored
from the branch-2 version, due to the differences in branch-2.8.
> Support user-specific minimum user limit percentage in Capacity Scheduler
> -
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Reporter: Eric Payne
>Assignee: Eric Payne
> Fix For: 3.0.0-alpha3
>
> Attachments: Active users highlighted.jpg, YARN-5892.001.patch,
> YARN-5892.002.patch, YARN-5892.003.patch, YARN-5892.004.patch,
> YARN-5892.005.patch, YARN-5892.006.patch, YARN-5892.007.patch,
> YARN-5892.008.patch, YARN-5892.009.patch, YARN-5892.010.patch,
> YARN-5892.012.patch, YARN-5892.013.patch, YARN-5892.014.patch,
> YARN-5892.015.patch, YARN-5892.branch-2.015.patch,
> YARN-5892.branch-2.016.patch, YARN-5892.branch-2.8.016.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
>
>
> yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent
> 25
>
>
>
> yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent
> 75
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6819) Application report fails if app rejected due to nodesize
[
https://issues.apache.org/jira/browse/YARN-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088699#comment-16088699
]
Hadoop QA commented on YARN-6819:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 42m 38s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6819 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877455/YARN-6819.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b44bb2a4acdd 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 06ece48 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/16456/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16456/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16456/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Application report fails if app rejected due to nodesize
>
>
> Key: YARN-6819
> URL:
[jira] [Updated] (YARN-6819) Application report fails if app rejected due to nodesize
[
https://issues.apache.org/jira/browse/YARN-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-6819:
---
Attachment: YARN-6819.003.patch
Thank you [~rohithsharma] for review .Attaching patch after handling checkstyle
> Application report fails if app rejected due to nodesize
>
>
> Key: YARN-6819
> URL: https://issues.apache.org/jira/browse/YARN-6819
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6819.001.patch, YARN-6819.002.patch,
> YARN-6819.003.patch
>
>
> In YARN-5006 application rejected when nodesize limit is exceeded.
> {{FinalSavingTransition}} stateBeforeFinalSaving not set after skipping save
> to store which causes application report failure
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3260) AM attempt fail to register before RM processes launch event
[ https://issues.apache.org/jira/browse/YARN-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088670#comment-16088670 ] Bibin A Chundatt commented on YARN-3260: Thank you [~jlowe] for review and commit. > AM attempt fail to register before RM processes launch event > > > Key: YARN-3260 > URL: https://issues.apache.org/jira/browse/YARN-3260 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Bibin A Chundatt >Priority: Critical > Fix For: 2.9.0, 2.7.4, 3.0.0-beta1, 2.8.2 > > Attachments: YARN-3260.001.patch > > > The RM on one of our clusters was running behind on processing > AsyncDispatcher events, and this caused AMs to fail to register due to an > NPE. The AM was launched and attempting to register before the > RMAppAttemptImpl had processed the LAUNCHED event, and the client to AM token > had not been generated yet. The NPE occurred because the > ApplicationMasterService tried to encode the missing token. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6816) Add functionality to remove labels from nodes
[ https://issues.apache.org/jira/browse/YARN-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088669#comment-16088669 ] Naganarasimha G R edited comment on YARN-6816 at 7/15/17 5:20 PM: -- As mentioned by [~sunilg] and [~bibinchundatt], there is a an option in the CLI to remove the labels on the nodes, hence i think as its invalid issue i am closing it, if you feel anything more can be done then we can reopen it. was (Author: naganarasimha): As mentioned by [~sunilg] and [~bibinchundatt], there is a an option in the CLI to remove the labels on the nodes, hence i think as its invalid issue i am closing it, if you feel. > Add functionality to remove labels from nodes > - > > Key: YARN-6816 > URL: https://issues.apache.org/jira/browse/YARN-6816 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung > > Seems in YARN-2505 functionality to remove labels from nodes was missed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6816) Add functionality to remove labels from nodes
[ https://issues.apache.org/jira/browse/YARN-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R resolved YARN-6816. - Resolution: Invalid As mentioned by [~sunilg] and [~bibinchundatt], there is a an option in the CLI to remove the labels on the nodes, hence i think as its invalid issue i am closing it, if you feel. > Add functionality to remove labels from nodes > - > > Key: YARN-6816 > URL: https://issues.apache.org/jira/browse/YARN-6816 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung > > Seems in YARN-2505 functionality to remove labels from nodes was missed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6826) SLS NMSimulator support for Opportunistic Container Queuing
[ https://issues.apache.org/jira/browse/YARN-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6826: -- Issue Type: Sub-task (was: Improvement) Parent: YARN-6828 > SLS NMSimulator support for Opportunistic Container Queuing > --- > > Key: YARN-6826 > URL: https://issues.apache.org/jira/browse/YARN-6826 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Arun Suresh >Assignee: Arun Suresh > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6828) [Umbrella] Container preemption using OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088662#comment-16088662 ] Arun Suresh commented on YARN-6828: --- Opening this as an umbrella JIRA as per [~leftnoteasy]'s request. Also created branch YARN-6828. > [Umbrella] Container preemption using OPPORTUNISTIC containers > -- > > Key: YARN-6828 > URL: https://issues.apache.org/jira/browse/YARN-6828 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > > This is based on discussions with [~kasha] and [~kkaranasos]. > Currently, the YARN schedulers selects containers for preemption only in > response to a starved queue / app's request. We propose to allow the > Schedulers to mark containers that are allocated over queue > capacity/fair-share as Opportunistic containers. > This JIRA proposes to allow Schedulers to: > # Allocate all containers over the configured queue capacity/weight as > OPPORTUNISTIC. > # Auto-promote running OPPORTUNISTIC containers of apps as and when their > GUARANTEED containers complete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6828) [Umbrella] Container preemption using OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6828: -- Description: This is based on discussions with [~kasha] and [~kkaranasos]. Currently, the YARN schedulers selects containers for preemption only in response to a starved queue / app's request. We propose to allow the Schedulers to mark containers that are allocated over queue capacity/fair-share as Opportunistic containers. This JIRA proposes to allow Schedulers to: # Allocate all containers over the configured queue capacity/weight as OPPORTUNISTIC. # Auto-promote running OPPORTUNISTIC containers of apps as and when their GUARANTEED containers complete. was: Currently, the YARN schedulers selects containers for preemption only in response to a starved queue / app's request. We propose to allow the Schedulers to mark containers that are allocated over queue capacity/fair-share as Opportunistic containers. This JIRA proposes to allow Schedulers to: # Allocate all containers over the configured queue capacity/weight as OPPORTUNISTIC. # Auto-promote running OPPORTUNISTIC containers of apps as and when their GUARANTEED containers complete. > [Umbrella] Container preemption using OPPORTUNISTIC containers > -- > > Key: YARN-6828 > URL: https://issues.apache.org/jira/browse/YARN-6828 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > > This is based on discussions with [~kasha] and [~kkaranasos]. > Currently, the YARN schedulers selects containers for preemption only in > response to a starved queue / app's request. We propose to allow the > Schedulers to mark containers that are allocated over queue > capacity/fair-share as Opportunistic containers. > This JIRA proposes to allow Schedulers to: > # Allocate all containers over the configured queue capacity/weight as > OPPORTUNISTIC. > # Auto-promote running OPPORTUNISTIC containers of apps as and when their > GUARANTEED containers complete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6829) Promote Opportunistic Containers to Guaranteed containers when Guaranteed containers complete
Arun Suresh created YARN-6829: - Summary: Promote Opportunistic Containers to Guaranteed containers when Guaranteed containers complete Key: YARN-6829 URL: https://issues.apache.org/jira/browse/YARN-6829 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun Suresh Assignee: Arun Suresh Once Guaranteed containers of apps complete, it is possible that the queue/app might go below configured capacity. In which case existing Opportunistic containers of an app can be promoted to ensure they are not preempted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6828) [Umbrella] Container preemption using OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6828: -- Description: Currently, the YARN schedulers selects containers for preemption only in response to a starved queue / app's request. We propose to allow the Schedulers to mark containers that are allocated over queue capacity/fair-share as Opportunistic containers. This JIRA proposes to allow Schedulers to: # Allocate all containers over the configured queue capacity/weight as OPPORTUNISTIC. # Auto-promote running OPPORTUNISTIC containers of apps as and when their GUARANTEED containers complete. was: Currently, the YARN schedulers selects containers for preemption only in response to a starved queue / app's request. We propose to allow the Schedulers to mark containers that are allocated over queue capacity/fair-share as Opportunistic containers. > [Umbrella] Container preemption using OPPORTUNISTIC containers > -- > > Key: YARN-6828 > URL: https://issues.apache.org/jira/browse/YARN-6828 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > > Currently, the YARN schedulers selects containers for preemption only in > response to a starved queue / app's request. We propose to allow the > Schedulers to mark containers that are allocated over queue > capacity/fair-share as Opportunistic containers. > This JIRA proposes to allow Schedulers to: > # Allocate all containers over the configured queue capacity/weight as > OPPORTUNISTIC. > # Auto-promote running OPPORTUNISTIC containers of apps as and when their > GUARANTEED containers complete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6828) [Umbrella] Container preemption using OPPORTUNISTIC containers
Arun Suresh created YARN-6828: - Summary: [Umbrella] Container preemption using OPPORTUNISTIC containers Key: YARN-6828 URL: https://issues.apache.org/jira/browse/YARN-6828 Project: Hadoop YARN Issue Type: Bug Reporter: Arun Suresh Assignee: Arun Suresh Currently, the YARN schedulers selects containers for preemption only in response to a starved queue / app's request. We propose to allow the Schedulers to mark containers that are allocated over queue capacity/fair-share as Opportunistic containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6808) Allow Schedulers to return OPPORTUNISTIC containers when queues go over configured capacity
[ https://issues.apache.org/jira/browse/YARN-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6808: -- Issue Type: Sub-task (was: New Feature) Parent: YARN-6828 > Allow Schedulers to return OPPORTUNISTIC containers when queues go over > configured capacity > --- > > Key: YARN-6808 > URL: https://issues.apache.org/jira/browse/YARN-6808 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6808.001.patch, YARN-6808.002.patch > > > This is based on discussions with [~kasha] and [~kkaranasos]. > Currently, when a Queues goes over capacity, apps on starved queues must wait > either for containers to complete or for them to be pre-empted by the > scheduler to get resources. > This JIRA proposes to allow Schedulers to: > # Allocate all containers over the configured queue capacity/weight as > OPPORTUNISTIC. > # Auto-promote running OPPORTUNISTIC containers of apps as and when their > GUARANTEED containers complete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6741) Deleting all children of a Parent Queue on refresh throws exception
[
https://issues.apache.org/jira/browse/YARN-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088599#comment-16088599
]
Naganarasimha G R commented on YARN-6741:
-
Thanks [~sunilg] for commenting on this issue.
The approach what is taken for deletion is first set the queue state to
*STOPPED* and then if the state of the queue after refresh has changed to
*STOPPED* (if running apps > 0 then queue's state is set to *DRAINING* ) then
we can proceed with deletion of the queue in the second refresh. With this we
ensure that the queue's are not deleted if it has running apps and admin
ensures that apps are either finished are killed before deletion of a queue.
But for the ParentQueue to be modified as LeafQueue, we do not have any
limitation of running apps and anyway if all the children are already deleted
then it implies that the running apps under this parentQueue is already zero,
So no need to ensure explicitly by first setting it as STOPPED and then
converting to LeafQueue.
> Deleting all children of a Parent Queue on refresh throws exception
> ---
>
> Key: YARN-6741
> URL: https://issues.apache.org/jira/browse/YARN-6741
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.0.0-alpha3
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-6741.001.patch, YARN-6741.002.patch,
> YARN-6741.003.patch
>
>
> If we configure CS such that all children of a parent queue are deleted and
> made as a leaf queue, then {{refreshQueue}} operation fails when
> re-initializing the parent Queue
> {code}
>// Sanity check
> if (!(newlyParsedQueue instanceof ParentQueue) || !newlyParsedQueue
> .getQueuePath().equals(getQueuePath())) {
> throw new IOException(
> "Trying to reinitialize " + getQueuePath() + " from "
> + newlyParsedQueue.getQueuePath());
> }
> {code}
> *Expected Behavior:*
> Converting a Parent Queue to leafQueue on refreshQueue needs to be supported.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6741) Deleting all children of a Parent Queue on refresh throws exception
[
https://issues.apache.org/jira/browse/YARN-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088497#comment-16088497
]
Sunil G commented on YARN-6741:
---
Thanks [~naganarasimha...@apache.org] for the patch
A quick question:
Does parent queue have to be stopped before converting it to a LeafQueue? (Or
before deleting all child queues) If the intention is to delete all children of
a queue, it could be STOPPED first and refreshQueue can be done. Post that
operation, we can start and start use it as LeafQueue. Please correct me if I
am wrong or missed something.
> Deleting all children of a Parent Queue on refresh throws exception
> ---
>
> Key: YARN-6741
> URL: https://issues.apache.org/jira/browse/YARN-6741
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.0.0-alpha3
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-6741.001.patch, YARN-6741.002.patch,
> YARN-6741.003.patch
>
>
> If we configure CS such that all children of a parent queue are deleted and
> made as a leaf queue, then {{refreshQueue}} operation fails when
> re-initializing the parent Queue
> {code}
>// Sanity check
> if (!(newlyParsedQueue instanceof ParentQueue) || !newlyParsedQueue
> .getQueuePath().equals(getQueuePath())) {
> throw new IOException(
> "Trying to reinitialize " + getQueuePath() + " from "
> + newlyParsedQueue.getQueuePath());
> }
> {code}
> *Expected Behavior:*
> Converting a Parent Queue to leafQueue on refreshQueue needs to be supported.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-4455) Support fetching metrics by time range
[ https://issues.apache.org/jira/browse/YARN-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088494#comment-16088494 ] Varun Saxena edited comment on YARN-4455 at 7/15/17 6:50 AM: - That should not be very difficult. Retrieving time series i.e. specifying metricstimestart and metricstimeend for last 24 hours should be as simple as getting current time(which would be in milliseconds assuming JAVA client) and deducting 24*60*60*1000 from it. The granularity at the level of millis or seconds would not be required but it should not be too cumbersome to convert it to millis even if specific date ranges are specified. Infact current time since epoch is more easily available. Moreover, created time range is also specified in milliseconds and so were time ranges specified in ATSv1 APIs'. was (Author: varun_saxena): Retrievi > Support fetching metrics by time range > -- > > Key: YARN-4455 > URL: https://issues.apache.org/jira/browse/YARN-4455 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-5355 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: YARN-4455-YARN-5355.01.patch, > YARN-4455-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4455) Support fetching metrics by time range
[ https://issues.apache.org/jira/browse/YARN-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088494#comment-16088494 ] Varun Saxena commented on YARN-4455: Retrievi > Support fetching metrics by time range > -- > > Key: YARN-4455 > URL: https://issues.apache.org/jira/browse/YARN-4455 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-5355 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: YARN-4455-YARN-5355.01.patch, > YARN-4455-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6741) Deleting all children of a Parent Queue on refresh throws exception
[
https://issues.apache.org/jira/browse/YARN-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088488#comment-16088488
]
Hadoop QA commented on YARN-6741:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 4 new + 222 unchanged - 2 fixed = 226 total (was 224)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 42m 21s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6741 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877438/YARN-6741.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e43b567d4238 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f413ee3 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16455/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/16455/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16455/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
[jira] [Updated] (YARN-6827) [ATS1/1.5] NPE exception while publishing recovering applications into ATS during RM restart.
[
https://issues.apache.org/jira/browse/YARN-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-6827:
Description:
While recovering application, it is observed that NPE exception is thrown as
below.
{noformat}
017-07-13 14:08:12,476 ERROR
org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV1Publisher:
Error when publishing entity [YARN_APPLICATION,application_1499929227397_0001]
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:178)
at
org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV1Publisher.putEntity(TimelineServiceV1Publisher.java:368)
{noformat}
This is because in RM service start, active services are started first in Non
HA case and later ATS services are started. In HA case, tansitionToActive event
has come first before ATS service are started.
This gives sufficient time to active services recover the applications which
tries to publish into ATS while recovering. Since ATS services are not started
yet, it throws NPE.
was:
While recovering application, it is observed that NPE exception is thrown as
below.
{noformat}
017-07-13 14:08:12,476 ERROR
org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV1Publisher:
Error when publishing entity [YARN_APPLICATION,application_1499929227397_0001]
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:178)
at
org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV1Publisher.putEntity(TimelineServiceV1Publisher.java:368)
{noformat}
This is because in RM service creation, active services are created first and
later ATS services are created. It means active services are started and ATS
services are started later point of time.
This gives sufficient time to active services recover the applications which
tries to publish into ATS while recovering. Since ATS services are not started
yet, it throws NPE.
> [ATS1/1.5] NPE exception while publishing recovering applications into ATS
> during RM restart.
> -
>
> Key: YARN-6827
> URL: https://issues.apache.org/jira/browse/YARN-6827
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>
> While recovering application, it is observed that NPE exception is thrown as
> below.
> {noformat}
> 017-07-13 14:08:12,476 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV1Publisher:
> Error when publishing entity
> [YARN_APPLICATION,application_1499929227397_0001]
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:178)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV1Publisher.putEntity(TimelineServiceV1Publisher.java:368)
> {noformat}
> This is because in RM service start, active services are started first in Non
> HA case and later ATS services are started. In HA case, tansitionToActive
> event has come first before ATS service are started.
> This gives sufficient time to active services recover the applications which
> tries to publish into ATS while recovering. Since ATS services are not
> started yet, it throws NPE.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
