[jira] [Commented] (YARN-998) Persistent resource change during NM/RM restart
[
https://issues.apache.org/jira/browse/YARN-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205907#comment-15205907
]
Hadoop QA commented on YARN-998:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
patch generated 0 new + 55 unchanged - 12 fixed = 55 total (was 67) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 57s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 21s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 171m 7s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.TestRMAdminService |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| JDK v1.7.0_95 Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerPreemption
|
| | hadoop.yarn.server.resourcema
[jira] [Created] (YARN-4852) Resource Manager Ran Out of Memory
Gokul created YARN-4852: --- Summary: Resource Manager Ran Out of Memory Key: YARN-4852 URL: https://issues.apache.org/jira/browse/YARN-4852 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.6.0 Reporter: Gokul Resource Manager went out of memory (max heap size: 8 GB, CMS GC) and shut down itself. Heap dump analysis reveals that 1200 instances of RMNodeImpl class hold 86% of memory. When digged deep, there are around 0.5 million objects of UpdatedContainerInfo (nodeUpdateQueue inside RMNodeImpl). This in turn contains around 1.7 million objects of YarnProtos$ContainerIdProto, ContainerStatusProto, ApplicationAttemptIdProto, ApplicationIdProto each of which retain around 1 GB heap. Full GC was triggered multiple times when RM went OOM and only 300 MB of heap was released. So all these objects look like live objects. RM's usual heap usage is around 4 GB but it suddenly spiked to 8 GB in 20 mins time and went OOM. There are no spike in job submissions, container numbers at the time of issue occurrence. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4517) [YARN-3368] Add nodes page
[ https://issues.apache.org/jira/browse/YARN-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205746#comment-15205746 ] Varun Saxena commented on YARN-4517: Thanks [~leftnoteasy] for the commit. Thanks [~gtCarrera9] and [~sunilg] for reviews. > [YARN-3368] Add nodes page > -- > > Key: YARN-4517 > URL: https://issues.apache.org/jira/browse/YARN-4517 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Wangda Tan >Assignee: Varun Saxena > Labels: webui > Fix For: YARN-3368 > > Attachments: (21-Feb-2016)yarn-ui-screenshots.zip, > Screenshot_after_4709.png, Screenshot_after_4709_1.png, > YARN-4517-YARN-3368.01.patch, YARN-4517-YARN-3368.02.patch > > > We need nodes page added to next generation web UI, similar to existing > RM/nodes page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[
https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205722#comment-15205722
]
Yi Zhou commented on YARN-4847:
---
Hi,
The created node labels disappear after restart resource manager(i have a node
label 'y'). It is a bug or limitation for node label feature ?
{code}
2016-03-22 11:31:28,936 FATAL
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
ResourceManager
org.apache.hadoop.service.ServiceStateException: java.io.IOException:
NodeLabelManager doesn't include label = y, please check.
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:569)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1000)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:262)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1221)
Caused by: java.io.IOException: NodeLabelManager doesn't include label = y,
please check.
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkIfLabelInClusterNodeLabels(SchedulerUtils.java:287)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:106)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:120)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:569)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:589)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:464)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:296)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:326)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 7 more
{code}
Thanks
> Add documentation for the Node Label features supported in 2.6
> ---
>
> Key: YARN-4847
> URL: https://issues.apache.org/jira/browse/YARN-4847
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.4
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>
> We constantly face issue with what are the node label supported features in
> 2.6 and general commands to use it. So it would be better to have
> documentation capturing what all is supported as part of 2.6
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3773) hadoop-yarn-server-nodemanager's use of Linux /sbin/tc is non-portable
[
https://issues.apache.org/jira/browse/YARN-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205680#comment-15205680
]
Sidharta Seethana commented on YARN-3773:
-
Hi [~alanburlison],
Could you please elaborate what you are referring to here?
{quote}
The code in question also seems to try to execute tc even on platforms where it
will never exist.
{quote}
The container-executor binary is only used by the LinuxContainerExecutor - so I
am wondering what you meant by this.
thanks,
-Sidharta
> hadoop-yarn-server-nodemanager's use of Linux /sbin/tc is non-portable
> --
>
> Key: YARN-3773
> URL: https://issues.apache.org/jira/browse/YARN-3773
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
> Environment: BSD OSX Solaris Windows Linux
>Reporter: Alan Burlison
>Assignee: Alan Burlison
>
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
> makes use of the Linux-only executable /sbin/tc
> (http://lartc.org/manpages/tc.txt) but there is no corresponding
> functionality for non-Linux platforms. The code in question also seems to try
> to execute tc even on platforms where it will never exist.
> Other platforms provide similar functionality, e.g. Solaris has an extensive
> range of network management features
> (http://www.oracle.com/technetwork/articles/servers-storage-admin/o11-095-s11-app-traffic-525038.html).
> Work is needed to abstract the network management features of Yarn so that
> the same facilities for network management can be provided on all platforms
> that provide the requisite functionality,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[
https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205677#comment-15205677
]
Yi Zhou commented on YARN-4847:
---
Hi,
I found if don't configure a 'default' queue with capacity in
capacity-scheduler.xml it will get failure when refresh queues(Restart resource
manager). I am not sure there is a limitation in CS of 2.6 ?
{code}
yarn.scheduler.capacity.root.queues
e,m,default
yarn.scheduler.capacity.root.default.capacity
25
{code}
> Add documentation for the Node Label features supported in 2.6
> ---
>
> Key: YARN-4847
> URL: https://issues.apache.org/jira/browse/YARN-4847
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.4
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>
> We constantly face issue with what are the node label supported features in
> 2.6 and general commands to use it. So it would be better to have
> documentation capturing what all is supported as part of 2.6
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205650#comment-15205650 ] Sunil G commented on YARN-4285: --- Hi [~jianhe], in YARN-4678, we were trying to take reserved capacity out of used capacity in UI. And given fox for same. Could you also pls take a look. > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4756) Unnecessary wait in Node Status Updater during reboot
[
https://issues.apache.org/jira/browse/YARN-4756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205613#comment-15205613
]
Hadoop QA commented on YARN-4756:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 58s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed with
JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 32s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed with
JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 18s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12794627/YARN-4756.003.patch |
| JIRA Issue | YARN-4756 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 682b9251c430 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e7ed05e |
| Defa
[jira] [Commented] (YARN-4657) Javadoc comment is broken for o.a.h.yarn.util.resource.Resources.multiplyByAndAddTo()
[
https://issues.apache.org/jira/browse/YARN-4657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205584#comment-15205584
]
Hadoop QA commented on YARN-4657:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 38s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 39s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 55s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12794611/YARN-4657.001.patch |
| JIRA Issue | YARN-4657 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8660ca28e21e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/pers
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205567#comment-15205567 ] Yi Zhou commented on YARN-4847: --- Add 5) There is no 'Node Labels' in left navigation on 'ResourceManager' UI even after enabling the node label feature in Hadoop 2.6 > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4711) NM is going down with NPE's due to single thread processing of events by Timeline client
[
https://issues.apache.org/jira/browse/YARN-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205510#comment-15205510
]
Sangjin Lee commented on YARN-4711:
---
Thanks for the proposed patch [~Naganarasimha]! I am going over it.
I did want to discuss one high level observation. It seems that you're taking
an approach of invoking the {{TimelineClient}} directly for async writes while
still using the dispatcher for sync writes. I understand that it is
functionally correct, and incidentally it also may solve one of the NPEs. On
the other hand, one downside is that we would have two very distinct sets of
code to write within {{NMTimelinePublisher}}, one for async writes and another
for sync writes. I'm still thinking about that, and I'm not sure whether it is
ideal or not.
If we had a way to address the NPE issue but stick with the current style
(using the dispatcher both for sync and async writes), it would lead to simpler
code that's easier to maintain, right? What is your thought on this? Pros and
cons?
> NM is going down with NPE's due to single thread processing of events by
> Timeline client
>
>
> Key: YARN-4711
> URL: https://issues.apache.org/jira/browse/YARN-4711
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Critical
> Labels: yarn-2928-1st-milestone
> Attachments: 4711Analysis.txt, YARN-4711-YARN-2928.v1.001.patch
>
>
> After YARN-3367, while testing the latest 2928 branch came across few NPEs
> due to which NM is shutting down.
> {code}
> 2016-02-21 23:19:54,078 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher:
> Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ContainerEventHandler.handle(NMTimelinePublisher.java:306)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ContainerEventHandler.handle(NMTimelinePublisher.java:296)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.putEntity(NMTimelinePublisher.java:213)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.publishContainerFinishedEvent(NMTimelinePublisher.java:192)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.access$400(NMTimelinePublisher.java:63)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ApplicationEventHandler.handle(NMTimelinePublisher.java:289)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ApplicationEventHandler.handle(NMTimelinePublisher.java:280)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> On analysis found that the there was delay in processing of events, as after
> YARN-3367 all the events were getting processed by a single thread inside the
> timeline client.
> Additionally found one scenario where there is possibility of NPE:
> * TimelineEntity.toString() when {{real}} is not null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[
https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205498#comment-15205498
]
Yi Zhou commented on YARN-4847:
---
Hi [~Naganarasimha], [~sunilg]
I summarized the below issues i came across in 2.6 so far..And i will make
progress to see others. Thank you for your patience !
1) Node label exclusivity is not supported like below command
sudo -u yarn yarn rmadmin -addToClusterNodeLabels "c(exclusive=false)"
addToClusterNodeLabels: java.io.IOException: label name should only
contains
{0-9, a-z, A-Z, -, _} and should not started with {-,_}, now it
is=c(exclusive=false)
2) Node labels are supported only in Capacity Scheduler
3) Listing node labels are not supported like below command
sudo -u yarn yarn cluster --list-node-labels
Error: Could not find or load main class cluster
4) Assign Node Labels to Cluster Node
Not supported format - sudo -u yarn yarn rmadmin -replaceLabelsOnNode
"bb-node3=y"
Supported format - sudo -u yarn yarn rmadmin -replaceLabelsOnNode
"bb-node3,y"
> Add documentation for the Node Label features supported in 2.6
> ---
>
> Key: YARN-4847
> URL: https://issues.apache.org/jira/browse/YARN-4847
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.4
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>
> We constantly face issue with what are the node label supported features in
> 2.6 and general commands to use it. So it would be better to have
> documentation capturing what all is supported as part of 2.6
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-998) Persistent resource change during NM/RM restart
[ https://issues.apache.org/jira/browse/YARN-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-998: Attachment: YARN-998-v3.patch Update the patch to address comments about concurrent issue and debug log. > Persistent resource change during NM/RM restart > --- > > Key: YARN-998 > URL: https://issues.apache.org/jira/browse/YARN-998 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-998-sample.patch, YARN-998-v1.patch, > YARN-998-v2.1.patch, YARN-998-v2.patch, YARN-998-v3.patch > > > When NM is restarted by plan or from a failure, previous dynamic resource > setting should be kept for consistency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4820) ResourceManager web redirects in HA mode drops query parameters
[
https://issues.apache.org/jira/browse/YARN-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205478#comment-15205478
]
Junping Du commented on YARN-4820:
--
Thanks [~vvasudev] for updating the patch!
The patch looks good in overall, just one comment in handling null charset case:
{code}
+ String reqEncoding = request.getCharacterEncoding();
+ if (reqEncoding == null || reqEncoding.isEmpty()) {
+reqEncoding = "UTF-8";
+ }
{code}
>From HTTP1.1 protocol:
>https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1, if no charset
>parameter is provided by request, we should treat it as "ISO-8859-1", all
>other charsets need to claim explicitly in request.
Other looks good to me.
> ResourceManager web redirects in HA mode drops query parameters
> ---
>
> Key: YARN-4820
> URL: https://issues.apache.org/jira/browse/YARN-4820
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-4820.001.patch, YARN-4820.002.patch
>
>
> The RMWebAppFilter redirects http requests from the standby to the active.
> However it drops all the query parameters when it does the redirect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4639) Remove dead code in TestDelegationTokenRenewer added in YARN-3055
[
https://issues.apache.org/jira/browse/YARN-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205448#comment-15205448
]
Hadoop QA commented on YARN-4639:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 4s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 16s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 21s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_95 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12794589/YARN-4639.001.patch |
| JIRA Issue | YARN-4639 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs c
[jira] [Commented] (YARN-998) Persistent resource change during NM/RM restart
[ https://issues.apache.org/jira/browse/YARN-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205383#comment-15205383 ] Junping Du commented on YARN-998: - Thanks [~jianhe] for comments! bq. should ResourceTrackerService#drConf be volatile? Nice catch! Will update it. bq. also, we may add a log in the resourceTrackerService where the resource is override to assist debugging. Agree. Will add a debug log. > Persistent resource change during NM/RM restart > --- > > Key: YARN-998 > URL: https://issues.apache.org/jira/browse/YARN-998 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-998-sample.patch, YARN-998-v1.patch, > YARN-998-v2.1.patch, YARN-998-v2.patch > > > When NM is restarted by plan or from a failure, previous dynamic resource > setting should be kept for consistency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4767) Network issues can cause persistent RM UI outage
[
https://issues.apache.org/jira/browse/YARN-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205374#comment-15205374
]
Hadoop QA commented on YARN-4767:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: patch
generated 3 new + 58 unchanged - 9 fixed = 61 total (was 67) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 15s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdk1.8.0_74
with JDK v1.8.0_74 generated 2 new + 98 unchanged - 2 fixed = 100 total (was
100) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hadoop-yarn-server-common in the patch passed with JDK
v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hadoop-yarn-server-web-proxy in the patch passed with
JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 0s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 2
[jira] [Created] (YARN-4851) Metric improvements for ATS v1.5 storage
Li Lu created YARN-4851: --- Summary: Metric improvements for ATS v1.5 storage Key: YARN-4851 URL: https://issues.apache.org/jira/browse/YARN-4851 Project: Hadoop YARN Issue Type: Sub-task Reporter: Li Lu Assignee: Li Lu We can add more metrics to the ATS v1.5 storage systems, including purging, cache hit/misses, read latency, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4711) NM is going down with NPE's due to single thread processing of events by Timeline client
[
https://issues.apache.org/jira/browse/YARN-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205334#comment-15205334
]
Hadoop QA commented on YARN-4711:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 20s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 42s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 9s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 55s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 17s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 35s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 2 new +
73 unchanged - 1 fixed = 75 total (was 74) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 11s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 2s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_74.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 58s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hadoop-yarn-server-common in the patch passed with JDK
v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 38s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:
[jira] [Updated] (YARN-4756) Unnecessary wait in Node Status Updater during reboot
[ https://issues.apache.org/jira/browse/YARN-4756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated YARN-4756: -- Attachment: YARN-4756.003.patch [~kasha], I wasn't clear in my original text. The patches in [YARN-4686] do not break any extra tests. However, while exploring the fixes for those failures, I came across an unnecessary wait in the NodeStatusUpdater thread, NodeStatusUpdaterImpl:850. When a reboot happens, the isStopped variable is set to true, but the thread waits until the next heartbeat. The next heartbeat won't come and so it will wait for a heartbeat timeout. So instead of wasting this time unnecessarily, I added a notify to wake the thread up and let it know to continue in the loop, where it would find that isStopped is set to true. Adding in this optimization uncovered a race condition in the TestNodeManagerResync test. The test doesn't wait for the NM to completely reboot before it checks for its updated capabilities. The only reason that it worked before is because the unnecessary wait in the NodeStatusUpdater acted as a sleep that masked the race condition. I'm uploading a patch that removes the unnecessary wait in the NodeStatusUpdater thread and also fixes the race condition in TestNodeManagerResync that it uncovers. > Unnecessary wait in Node Status Updater during reboot > - > > Key: YARN-4756 > URL: https://issues.apache.org/jira/browse/YARN-4756 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: YARN-4756.001.patch, YARN-4756.002.patch, > YARN-4756.003.patch > > > The startStatusUpdater thread waits for the isStopped variable to be set to > true, but it is waiting for the next heartbeat. During a reboot, the next > heartbeat will not come and so the thread waits for a timeout. Instead, we > should notify the thread to continue so that it can check the isStopped > variable and exit without having to wait for a timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4756) Unnecessary wait in Node Status Updater during reboot
[ https://issues.apache.org/jira/browse/YARN-4756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205278#comment-15205278 ] Karthik Kambatla commented on YARN-4756: The patch doesn't apply anymore. Can we update this? Also, we should avoid putting changes into trunk and branch-* when we know the patch is breaking tests. > Unnecessary wait in Node Status Updater during reboot > - > > Key: YARN-4756 > URL: https://issues.apache.org/jira/browse/YARN-4756 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: YARN-4756.001.patch, YARN-4756.002.patch > > > The startStatusUpdater thread waits for the isStopped variable to be set to > true, but it is waiting for the next heartbeat. During a reboot, the next > heartbeat will not come and so the thread waits for a timeout. Instead, we > should notify the thread to continue so that it can check the isStopped > variable and exit without having to wait for a timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205262#comment-15205262 ] Jian He commented on YARN-4285: --- [~vvasudev], [~leftnoteasy], looks like the % of queue/cluster for the app does not include reserved resources. However, the queue's used resource in the UI does include reserved resource too. This caused the total amount of resources for all apps in the table does not equal to the total used resource shown in the queue. Should we consider reserved resource too in the % of queue/cluster for an app ? > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-4850) test-fair-scheduler.xml isn't valid xml
[ https://issues.apache.org/jira/browse/YARN-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu reassigned YARN-4850: -- Assignee: Yufei Gu > test-fair-scheduler.xml isn't valid xml > --- > > Key: YARN-4850 > URL: https://issues.apache.org/jira/browse/YARN-4850 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, test >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Yufei Gu >Priority: Blocker > > The ASF license should be in an actual XML-formatted comment inside the XML > block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4843) [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to int64
[ https://issues.apache.org/jira/browse/YARN-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205205#comment-15205205 ] Jian He commented on YARN-4843: --- can we change the existing int field to long, instead of adding a new field ? Setter API is fine as long can accept int. Need a new getter API to return the long type. > [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to > int64 > - > > Key: YARN-4843 > URL: https://issues.apache.org/jira/browse/YARN-4843 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Reporter: Wangda Tan > > This JIRA is to track all int32 usages in YARN's ProtocolBuffer APIs that we > possibly need to update to int64. > One example is resource API. We use int32 for memory now, if a cluster has > 10k nodes, each node has 210G memory, we will get a negative total cluster > memory. > We may have other fields may need to upgrade from int32 to int64. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-998) Persistent resource change during NM/RM restart
[
https://issues.apache.org/jira/browse/YARN-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205193#comment-15205193
]
Jian He commented on YARN-998:
--
looks good to me overall,
- should ResourceTrackerService#drConf be volatile ?
- also, we may add a log in the resourceTrackerService where the resource is
override to assist debugging.
{code}
if (nodes != null && Arrays.asList(nodes).contains(nid)) {
capability.setMemory(this.drConf.getMemoryPerNode(nid));
capability.setVirtualCores(this.drConf.getVcoresPerNode(nid));
}
{code}
> Persistent resource change during NM/RM restart
> ---
>
> Key: YARN-998
> URL: https://issues.apache.org/jira/browse/YARN-998
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: graceful, nodemanager, scheduler
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: YARN-998-sample.patch, YARN-998-v1.patch,
> YARN-998-v2.1.patch, YARN-998-v2.patch
>
>
> When NM is restarted by plan or from a failure, previous dynamic resource
> setting should be kept for consistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4657) Javadoc comment is broken for o.a.h.yarn.util.resource.Resources.multiplyByAndAddTo()
[
https://issues.apache.org/jira/browse/YARN-4657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-4657:
---
Attachment: YARN-4657.001.patch
> Javadoc comment is broken for
> o.a.h.yarn.util.resource.Resources.multiplyByAndAddTo()
> -
>
> Key: YARN-4657
> URL: https://issues.apache.org/jira/browse/YARN-4657
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Attachments: YARN-4657.001.patch
>
>
> The comment is
> {code}
> /**
>* Multiply @param rhs by @param by, and add the result to @param lhs
>* without creating any new {@link Resource} object
>*/
> {code}
> The {{@param}} tag can't be used that way. {{\{@code rhs\}}} is the correct
> thing to do.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-4640) Replace synchronized HashSet with something from util.concurrent in DelegationTokenRenewer
[
https://issues.apache.org/jira/browse/YARN-4640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton resolved YARN-4640.
Resolution: Invalid
> Replace synchronized HashSet with something from util.concurrent in
> DelegationTokenRenewer
> --
>
> Key: YARN-4640
> URL: https://issues.apache.org/jira/browse/YARN-4640
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>
> {code}
> this.referringAppIds = Collections.synchronizedSet(
> new HashSet(applicationIds));
> {code}
> should probably be a {{ConcurrentHashMap}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4640) Replace synchronized HashSet with something from util.concurrent in DelegationTokenRenewer
[
https://issues.apache.org/jira/browse/YARN-4640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-4640:
---
Summary: Replace synchronized HashSet with something from util.concurrent
in DelegationTokenRenewer (was: Replace synchronized HashSet with something
from util.concurrent in DelegationTkenRenewer)
> Replace synchronized HashSet with something from util.concurrent in
> DelegationTokenRenewer
> --
>
> Key: YARN-4640
> URL: https://issues.apache.org/jira/browse/YARN-4640
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>
> {code}
> this.referringAppIds = Collections.synchronizedSet(
> new HashSet(applicationIds));
> {code}
> should probably be a {{ConcurrentHashMap}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-4787) Remove dead readObject() method from WebAppProxyServet
[ https://issues.apache.org/jira/browse/YARN-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton resolved YARN-4787. Resolution: Won't Fix The comments on the method explain why it cannot be removed. > Remove dead readObject() method from WebAppProxyServet > -- > > Key: YARN-4787 > URL: https://issues.apache.org/jira/browse/YARN-4787 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Attachments: YARN-4787.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4787) Remove dead readObject() method from WebAppProxyServet
[ https://issues.apache.org/jira/browse/YARN-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4787: --- Attachment: YARN-4787.001.patch > Remove dead readObject() method from WebAppProxyServet > -- > > Key: YARN-4787 > URL: https://issues.apache.org/jira/browse/YARN-4787 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Attachments: YARN-4787.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-998) Persistent resource change during NM/RM restart
[
https://issues.apache.org/jira/browse/YARN-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205065#comment-15205065
]
Hadoop QA commented on YARN-998:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
patch generated 0 new + 54 unchanged - 12 fixed = 54 total (was 66) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 13s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 14s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 51s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestRMAdminService |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_95 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestRMAdminService |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| D
[jira] [Updated] (YARN-4850) test-fair-scheduler.xml isn't valid xml
[ https://issues.apache.org/jira/browse/YARN-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-4850: --- Description: The ASF license should be in an actual XML-formatted comment inside the XML block. (was: ) > test-fair-scheduler.xml isn't valid xml > --- > > Key: YARN-4850 > URL: https://issues.apache.org/jira/browse/YARN-4850 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, test >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Priority: Blocker > > The ASF license should be in an actual XML-formatted comment inside the XML > block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4850) test-fair-scheduler.xml isn't valid xml
Allen Wittenauer created YARN-4850: -- Summary: test-fair-scheduler.xml isn't valid xml Key: YARN-4850 URL: https://issues.apache.org/jira/browse/YARN-4850 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Priority: Blocker -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4849) [YARN-3368] cleanup code base, integrate web UI related build to mvn, and add licenses.
Wangda Tan created YARN-4849: Summary: [YARN-3368] cleanup code base, integrate web UI related build to mvn, and add licenses. Key: YARN-4849 URL: https://issues.apache.org/jira/browse/YARN-4849 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4807) MockAM#waitForState sleep duration is too long
[
https://issues.apache.org/jira/browse/YARN-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205047#comment-15205047
]
Yufei Gu commented on YARN-4807:
1. {quote}I have seen test cases where there is no reference to an RM or
MockRM.{quote}
I've checked all usages of {{waitForState}} in {{MockAM}}. All no reference to
an RM or MockRm places are inside the class {{MockAM}}. So that, we can make
{{waitForState}} in {{MockAM}} private, and change all outside reference to
{{waitForState}} in {{MockRM}}.
2. I agree to have some constants for {{waitMsPerLoop}}, {{minWaitMsecs}} and
{{timeoutMsecs}}, shared by {{MockAM}} and {{MockRM}}.
> MockAM#waitForState sleep duration is too long
> --
>
> Key: YARN-4807
> URL: https://issues.apache.org/jira/browse/YARN-4807
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Yufei Gu
> Labels: newbie
>
> MockAM#waitForState sleep duration (500 ms) is too long. Also, there is
> significant duplication with MockRM#waitForState.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4711) NM is going down with NPE's due to single thread processing of events by Timeline client
[
https://issues.apache.org/jira/browse/YARN-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-4711:
Attachment: YARN-4711-YARN-2928.v1.001.patch
hi [~sjlee0],
I have incorporated the changes as per the discussion. please review
> NM is going down with NPE's due to single thread processing of events by
> Timeline client
>
>
> Key: YARN-4711
> URL: https://issues.apache.org/jira/browse/YARN-4711
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Critical
> Labels: yarn-2928-1st-milestone
> Attachments: 4711Analysis.txt, YARN-4711-YARN-2928.v1.001.patch
>
>
> After YARN-3367, while testing the latest 2928 branch came across few NPEs
> due to which NM is shutting down.
> {code}
> 2016-02-21 23:19:54,078 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher:
> Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ContainerEventHandler.handle(NMTimelinePublisher.java:306)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ContainerEventHandler.handle(NMTimelinePublisher.java:296)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.putEntity(NMTimelinePublisher.java:213)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.publishContainerFinishedEvent(NMTimelinePublisher.java:192)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.access$400(NMTimelinePublisher.java:63)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ApplicationEventHandler.handle(NMTimelinePublisher.java:289)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ApplicationEventHandler.handle(NMTimelinePublisher.java:280)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> On analysis found that the there was delay in processing of events, as after
> YARN-3367 all the events were getting processed by a single thread inside the
> timeline client.
> Additionally found one scenario where there is possibility of NPE:
> * TimelineEntity.toString() when {{real}} is not null
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4517) [YARN-3368] Add nodes page
[ https://issues.apache.org/jira/browse/YARN-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205027#comment-15205027 ] Wangda Tan commented on YARN-4517: -- Committed to branch:YARN-3368, tested it locally before push. > [YARN-3368] Add nodes page > -- > > Key: YARN-4517 > URL: https://issues.apache.org/jira/browse/YARN-4517 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Wangda Tan >Assignee: Varun Saxena > Labels: webui > Fix For: YARN-3368 > > Attachments: (21-Feb-2016)yarn-ui-screenshots.zip, > Screenshot_after_4709.png, Screenshot_after_4709_1.png, > YARN-4517-YARN-3368.01.patch, YARN-4517-YARN-3368.02.patch > > > We need nodes page added to next generation web UI, similar to existing > RM/nodes page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4517) [YARN-3368] Add nodes page
[ https://issues.apache.org/jira/browse/YARN-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205003#comment-15205003 ] Wangda Tan commented on YARN-4517: -- +1 to latest patch, will commit to branch soon to unblock YARN-4734. Thanks [~varun_saxena] and reviews from [~sunilg]/[~gtCarrera9]. > [YARN-3368] Add nodes page > -- > > Key: YARN-4517 > URL: https://issues.apache.org/jira/browse/YARN-4517 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Wangda Tan >Assignee: Varun Saxena > Labels: webui > Attachments: (21-Feb-2016)yarn-ui-screenshots.zip, > Screenshot_after_4709.png, Screenshot_after_4709_1.png, > YARN-4517-YARN-3368.01.patch, YARN-4517-YARN-3368.02.patch > > > We need nodes page added to next generation web UI, similar to existing > RM/nodes page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4639) Remove dead code in TestDelegationTokenRenewer added in YARN-3055
[
https://issues.apache.org/jira/browse/YARN-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-4639:
---
Attachment: YARN-4639.001.patch
> Remove dead code in TestDelegationTokenRenewer added in YARN-3055
> -
>
> Key: YARN-4639
> URL: https://issues.apache.org/jira/browse/YARN-4639
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Attachments: YARN-4639.001.patch
>
>
> Remove lines 1093-1094:
> {code}
> //MyFS fs = (MyFS)FileSystem.get(conf);
> //MyToken token1 = fs.getDelegationToken("user123");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4767) Network issues can cause persistent RM UI outage
[ https://issues.apache.org/jira/browse/YARN-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4767: --- Attachment: YARN-4767.004.patch > Network issues can cause persistent RM UI outage > > > Key: YARN-4767 > URL: https://issues.apache.org/jira/browse/YARN-4767 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-4767.001.patch, YARN-4767.002.patch, > YARN-4767.003.patch, YARN-4767.004.patch > > > If a network issue causes an AM web app to resolve the RM proxy's address to > something other than what's listed in the allowed proxies list, the > AmIpFilter will 302 redirect the RM proxy's request back to the RM proxy. > The RM proxy will then consume all available handler threads connecting to > itself over and over, resulting in an outage of the web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4767) Network issues can cause persistent RM UI outage
[ https://issues.apache.org/jira/browse/YARN-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4767: --- Attachment: (was: YARN-4767.004.patch) > Network issues can cause persistent RM UI outage > > > Key: YARN-4767 > URL: https://issues.apache.org/jira/browse/YARN-4767 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-4767.001.patch, YARN-4767.002.patch, > YARN-4767.003.patch > > > If a network issue causes an AM web app to resolve the RM proxy's address to > something other than what's listed in the allowed proxies list, the > AmIpFilter will 302 redirect the RM proxy's request back to the RM proxy. > The RM proxy will then consume all available handler threads connecting to > itself over and over, resulting in an outage of the web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4767) Network issues can cause persistent RM UI outage
[ https://issues.apache.org/jira/browse/YARN-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4767: --- Attachment: YARN-4767.004.patch > Network issues can cause persistent RM UI outage > > > Key: YARN-4767 > URL: https://issues.apache.org/jira/browse/YARN-4767 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-4767.001.patch, YARN-4767.002.patch, > YARN-4767.003.patch, YARN-4767.004.patch > > > If a network issue causes an AM web app to resolve the RM proxy's address to > something other than what's listed in the allowed proxies list, the > AmIpFilter will 302 redirect the RM proxy's request back to the RM proxy. > The RM proxy will then consume all available handler threads connecting to > itself over and over, resulting in an outage of the web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4767) Network issues can cause persistent RM UI outage
[ https://issues.apache.org/jira/browse/YARN-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4767: --- Attachment: (was: YARN-4767.004.patch) > Network issues can cause persistent RM UI outage > > > Key: YARN-4767 > URL: https://issues.apache.org/jira/browse/YARN-4767 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-4767.001.patch, YARN-4767.002.patch, > YARN-4767.003.patch, YARN-4767.004.patch > > > If a network issue causes an AM web app to resolve the RM proxy's address to > something other than what's listed in the allowed proxies list, the > AmIpFilter will 302 redirect the RM proxy's request back to the RM proxy. > The RM proxy will then consume all available handler threads connecting to > itself over and over, resulting in an outage of the web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4767) Network issues can cause persistent RM UI outage
[ https://issues.apache.org/jira/browse/YARN-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4767: --- Attachment: YARN-4767.004.patch This patch resolves the checkstyle issues that can be resolved and fixes the test failures. > Network issues can cause persistent RM UI outage > > > Key: YARN-4767 > URL: https://issues.apache.org/jira/browse/YARN-4767 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-4767.001.patch, YARN-4767.002.patch, > YARN-4767.003.patch, YARN-4767.004.patch > > > If a network issue causes an AM web app to resolve the RM proxy's address to > something other than what's listed in the allowed proxies list, the > AmIpFilter will 302 redirect the RM proxy's request back to the RM proxy. > The RM proxy will then consume all available handler threads connecting to > itself over and over, resulting in an outage of the web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4781) Support intra-queue preemption for fairness ordering policy.
[ https://issues.apache.org/jira/browse/YARN-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204957#comment-15204957 ] Miles Crawford commented on YARN-4781: -- This is important for our organization. Currently, if a job in queue A is using 100% of the cluster resources, and a new job arrives in queue A, it sometimes cannot even get an application master! Two jobs in queue A should share their resources fairly with pre-emption. It should be feasible to use the same preemption rules for contention within a queue as for contention across queues. We have to work around this currently by giving every job a unique name - for example, we submit jobs to queue "A-2016-03-21T08:23". Then each job is in its own queue, and can use preemption to make progress. > Support intra-queue preemption for fairness ordering policy. > > > Key: YARN-4781 > URL: https://issues.apache.org/jira/browse/YARN-4781 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > > We introduced fairness queue policy since YARN-3319, which will let large > applications make progresses and not starve small applications. However, if a > large application takes the queue’s resources, and containers of the large > app has long lifespan, small applications could still wait for resources for > long time and SLAs cannot be guaranteed. > Instead of wait for application release resources on their own, we need to > preempt resources of queue with fairness policy enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4517) [YARN-3368] Add nodes page
[
https://issues.apache.org/jira/browse/YARN-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204878#comment-15204878
]
Wangda Tan commented on YARN-4517:
--
[~varun_saxena],
bq. With aggregation disabled ? Are you testing with latest trunk code ? We had
issues before YARN-4709 went in.
It's not latest trunk, I will try this on latest trunk.
bq. Ok. Will check. Even with full(not minimized) browser window ?
Yes
bq. NodeId is required to query node info from RM. It has been passed around
everywhere because we need to create link for left hand side menu on every page.
Look closer to URLs. First, you can move the left side menu to yarn-node.hbs.
Url can be generated by model fields like {{model.rmNode.id}}, I think you
don't need a separate piece in URL like {{localhost:61236}}
And for urls of nodes, I would suggest update URLs to:
{code}
../yarnNodes/
{code}
For overview of nodes (node information)
{code}
../yarnNodes//runningApps
{code}
For running applications
{code}
../yarnNodes//runningContainers
{code}
For running containers.
These items could be fixed together in YARN-4848.
bq. Do we need AM pages too for merge to trunk ?
We may not need it. Let's try to merge infra code to trunk first.
> [YARN-3368] Add nodes page
> --
>
> Key: YARN-4517
> URL: https://issues.apache.org/jira/browse/YARN-4517
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Wangda Tan
>Assignee: Varun Saxena
> Labels: webui
> Attachments: (21-Feb-2016)yarn-ui-screenshots.zip,
> Screenshot_after_4709.png, Screenshot_after_4709_1.png,
> YARN-4517-YARN-3368.01.patch, YARN-4517-YARN-3368.02.patch
>
>
> We need nodes page added to next generation web UI, similar to existing
> RM/nodes page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4756) Unnecessary wait in Node Status Updater during reboot
[ https://issues.apache.org/jira/browse/YARN-4756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204875#comment-15204875 ] Eric Badger commented on YARN-4756: --- [~eepayne], [~jlowe], [~kasha], can one of you review this patch when you are available? This patch is a byproduct of test failures that were uncovered when fixing [YARN-4686]. Thanks. > Unnecessary wait in Node Status Updater during reboot > - > > Key: YARN-4756 > URL: https://issues.apache.org/jira/browse/YARN-4756 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: YARN-4756.001.patch, YARN-4756.002.patch > > > The startStatusUpdater thread waits for the isStopped variable to be set to > true, but it is waiting for the next heartbeat. During a reboot, the next > heartbeat will not come and so the thread waits for a timeout. Instead, we > should notify the thread to continue so that it can check the isStopped > variable and exit without having to wait for a timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4815) ATS 1.5 timelineclinet impl try to create attempt directory for every event call
[
https://issues.apache.org/jira/browse/YARN-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204786#comment-15204786
]
Hadoop QA commented on YARN-4815:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 6s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 2s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 39s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_74.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 56s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}
[jira] [Commented] (YARN-998) Persistent resource change during NM/RM restart
[ https://issues.apache.org/jira/browse/YARN-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204769#comment-15204769 ] Junping Du commented on YARN-998: - Verified in locally that TestRMAdminService.testResourcePersistentForNMRegistrationWithNewResource() is successful. Rerun the test again. > Persistent resource change during NM/RM restart > --- > > Key: YARN-998 > URL: https://issues.apache.org/jira/browse/YARN-998 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-998-sample.patch, YARN-998-v1.patch, > YARN-998-v2.1.patch, YARN-998-v2.patch > > > When NM is restarted by plan or from a failure, previous dynamic resource > setting should be kept for consistency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4685) AM blacklisting result in application to get hanged
[
https://issues.apache.org/jira/browse/YARN-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204712#comment-15204712
]
Sunil G commented on YARN-4685:
---
Agreeing to your point [~rohithsharma].
We have {{blacklistManager}} per {{RMAppAttempt}}. So to operate anything on
{{blacklistManager}}, we have to pass reference to scheduler. Assuming I am
interested in your second approach. In Each heartbeat call, we will check for
pending AM container resource request. Then for such resource request,
re-compute blacklist threshold if needed (which means if some nodes are
added/removed recently) in {{blacklistManager}}. If there are some changes in
threshold, remove blacklist for this ResourceRequest.
But we need to change lot of interface api syntax. If we had a common
BlackListManager, which keeps tracks of all blacklist information for all apps,
it would have been more clean.
> AM blacklisting result in application to get hanged
> ---
>
> Key: YARN-4685
> URL: https://issues.apache.org/jira/browse/YARN-4685
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>
> AM blacklist addition or removal is updated only when RMAppAttempt is
> scheduled i.e {{RMAppAttemptImpl#ScheduleTransition#transition}}. But once
> attempt is scheduled if there is any removeNode/addNode in cluster then this
> is not updated to {{BlackListManager#refreshNodeHostCount}}. This leads
> BlackListManager to operate on stale NM's count. And application is in
> ACCEPTED state and wait forever even if we add more nodes to cluster.
> Solution is update BlacklistManager for every
> {{RMAppAttemptImpl#AMContainerAllocatedTransition#transition}} call. This
> ensures if there is any addition/removal in nodes, this will be updated to
> BlacklistManager
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4843) [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to int64
[ https://issues.apache.org/jira/browse/YARN-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204654#comment-15204654 ] Karthik Kambatla commented on YARN-4843: Sure, all client-facing updates need to be compatible. If we are doing it only in trunk, we could at least do the updates between NM-RM incompatibly. > [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to > int64 > - > > Key: YARN-4843 > URL: https://issues.apache.org/jira/browse/YARN-4843 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Reporter: Wangda Tan > > This JIRA is to track all int32 usages in YARN's ProtocolBuffer APIs that we > possibly need to update to int64. > One example is resource API. We use int32 for memory now, if a cluster has > 10k nodes, each node has 210G memory, we will get a negative total cluster > memory. > We may have other fields may need to upgrade from int32 to int64. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4685) AM blacklisting result in application to get hanged
[
https://issues.apache.org/jira/browse/YARN-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204685#comment-15204685
]
Rohith Sharma K S commented on YARN-4685:
-
Initially thought to fix by calling another allocate call when ever there is
node update event to
{{RMApp->RMAppImpl}}. But there could be case where newly allocate call get the
master container before RMAppAttemptImpl gets container allocated event. In
such case, RMAppAttemptImpl should have handling mechanism. Like this many
cases can occur. This option does not work.
Other approaches fixing this issue are recompute blacklist threshold EITHER for
on node-added && node-remove event OR on every heartbeat for the *ALL* apps
which are waiting for AM container allocation and update appschedulinginfo for
{{amBlacklist}}
> AM blacklisting result in application to get hanged
> ---
>
> Key: YARN-4685
> URL: https://issues.apache.org/jira/browse/YARN-4685
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>
> AM blacklist addition or removal is updated only when RMAppAttempt is
> scheduled i.e {{RMAppAttemptImpl#ScheduleTransition#transition}}. But once
> attempt is scheduled if there is any removeNode/addNode in cluster then this
> is not updated to {{BlackListManager#refreshNodeHostCount}}. This leads
> BlackListManager to operate on stale NM's count. And application is in
> ACCEPTED state and wait forever even if we add more nodes to cluster.
> Solution is update BlacklistManager for every
> {{RMAppAttemptImpl#AMContainerAllocatedTransition#transition}} call. This
> ensures if there is any addition/removal in nodes, this will be updated to
> BlacklistManager
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204656#comment-15204656 ] Wangda Tan commented on YARN-796: - [~jameszhouyi], bq. Illegal capacity of 1.0 for children of queue root for label=y You should set root.accessible-node-labels.y.capacity = 100. And Node Labels UI may not be existed in 2.6. I would suggest you send more question about usages of YARN components to hadoop users mail list. JIRA is majorly for dev discussions. Thanks, > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-998) Persistent resource change during NM/RM restart
[
https://issues.apache.org/jira/browse/YARN-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204666#comment-15204666
]
Hadoop QA commented on YARN-998:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
patch generated 0 new + 55 unchanged - 12 fixed = 55 total (was 67) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 30s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 7s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 186m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling |
| | hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.TestRMAdminService |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| JDK
[jira] [Commented] (YARN-4842) yarn logs command should not require the appOwner argument
[
https://issues.apache.org/jira/browse/YARN-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204647#comment-15204647
]
Hadoop QA commented on YARN-4842:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 6s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 31s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 2 new +
50 unchanged - 1 fixed = 52 total (was 51) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 54s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 18s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_74.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 10s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 34s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 158m 55s {c
[jira] [Commented] (YARN-4843) [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to int64
[ https://issues.apache.org/jira/browse/YARN-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204621#comment-15204621 ] Wangda Tan commented on YARN-4843: -- [~kasha], actually we can do this in compatible way like what [~djp] suggested: add two int64 fields, deprecate int32 fields. User can set/get both fields, but int64 fields will be preferred. I would suggest to keep compatibility even we ship 3.0.0, all YARN applications use the Resource object frequently, it gonna be hard to ask everyone update lots of source code to use 3.0.0. > [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to > int64 > - > > Key: YARN-4843 > URL: https://issues.apache.org/jira/browse/YARN-4843 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Reporter: Wangda Tan > > This JIRA is to track all int32 usages in YARN's ProtocolBuffer APIs that we > possibly need to update to int64. > One example is resource API. We use int32 for memory now, if a cluster has > 10k nodes, each node has 210G memory, we will get a negative total cluster > memory. > We may have other fields may need to upgrade from int32 to int64. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4843) [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to int64
[ https://issues.apache.org/jira/browse/YARN-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204587#comment-15204587 ] Karthik Kambatla commented on YARN-4843: Since 3.0.0 is not too far out, I would like for us to update on trunk. /cc [~andrew.wang] > [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to > int64 > - > > Key: YARN-4843 > URL: https://issues.apache.org/jira/browse/YARN-4843 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Reporter: Wangda Tan > > This JIRA is to track all int32 usages in YARN's ProtocolBuffer APIs that we > possibly need to update to int64. > One example is resource API. We use int32 for memory now, if a cluster has > 10k nodes, each node has 210G memory, we will get a negative total cluster > memory. > We may have other fields may need to upgrade from int32 to int64. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4843) [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to int64
[ https://issues.apache.org/jira/browse/YARN-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4843: --- Hadoop Flags: Incompatible change > [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to > int64 > - > > Key: YARN-4843 > URL: https://issues.apache.org/jira/browse/YARN-4843 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Reporter: Wangda Tan > > This JIRA is to track all int32 usages in YARN's ProtocolBuffer APIs that we > possibly need to update to int64. > One example is resource API. We use int32 for memory now, if a cluster has > 10k nodes, each node has 210G memory, we will get a negative total cluster > memory. > We may have other fields may need to upgrade from int32 to int64. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4843) [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to int64
[ https://issues.apache.org/jira/browse/YARN-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4843: --- Target Version/s: 3.0.0 > [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to > int64 > - > > Key: YARN-4843 > URL: https://issues.apache.org/jira/browse/YARN-4843 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Reporter: Wangda Tan > > This JIRA is to track all int32 usages in YARN's ProtocolBuffer APIs that we > possibly need to update to int64. > One example is resource API. We use int32 for memory now, if a cluster has > 10k nodes, each node has 210G memory, we will get a negative total cluster > memory. > We may have other fields may need to upgrade from int32 to int64. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4815) ATS 1.5 timelineclinet impl try to create attempt directory for every event call
[ https://issues.apache.org/jira/browse/YARN-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-4815: Attachment: YARN-4815.3.patch > ATS 1.5 timelineclinet impl try to create attempt directory for every event > call > > > Key: YARN-4815 > URL: https://issues.apache.org/jira/browse/YARN-4815 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4815.1.patch, YARN-4815.2.patch, YARN-4815.3.patch > > > ATS 1.5 timelineclinet impl, try to create attempt directory for every event > call. Since per attempt only one call to create directory is enough, this is > causing perf issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4815) ATS 1.5 timelineclinet impl try to create attempt directory for every event call
[ https://issues.apache.org/jira/browse/YARN-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204552#comment-15204552 ] Xuan Gong commented on YARN-4815: - rebase the patch to fix the checkstyle issues. > ATS 1.5 timelineclinet impl try to create attempt directory for every event > call > > > Key: YARN-4815 > URL: https://issues.apache.org/jira/browse/YARN-4815 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4815.1.patch, YARN-4815.2.patch, YARN-4815.3.patch > > > ATS 1.5 timelineclinet impl, try to create attempt directory for every event > call. Since per attempt only one call to create directory is enough, this is > causing perf issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204471#comment-15204471 ] Sunil G commented on YARN-4847: --- Yes. It's perfectly fine. Later we can point to the discussions here also. No issues :-) > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204465#comment-15204465 ] Naganarasimha G R commented on YARN-4847: - bq. if it's more of installation and use case pblm, it can be put in dev mailing list also Yes i agree its more of installation and usage problem, but the concern is might not be all member remember to search the mailing list for the same and ofcourse we too do find it confusing whats supported in 2.6 and 2.7 so better to have documentation in 2.6 so that its easy to refer back to. Thoughts ? > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204452#comment-15204452 ] Sunil G commented on YARN-4847: --- I think if it's more of installation and use case pblm, it can be put in dev mailing list also. But if it has more input towards documentation, this ticket ll be also a correct place. > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1520#comment-1520 ] Naganarasimha G R commented on YARN-4847: - Hi [~jameszhouyi], Lets keep more on discussions with 2.6 Node label features in this jira than YARN-796 as it would clutter there. > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4686) MiniYARNCluster.start() returns before cluster is completely started
[
https://issues.apache.org/jira/browse/YARN-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204435#comment-15204435
]
Eric Badger commented on YARN-4686:
---
Thanks, [~eepayne]!
> MiniYARNCluster.start() returns before cluster is completely started
>
>
> Key: YARN-4686
> URL: https://issues.apache.org/jira/browse/YARN-4686
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Eric Badger
> Fix For: 2.7.3
>
> Attachments: MAPREDUCE-6507.001.patch,
> YARN-4686-branch-2.7.006.patch, YARN-4686.001.patch, YARN-4686.002.patch,
> YARN-4686.003.patch, YARN-4686.004.patch, YARN-4686.005.patch,
> YARN-4686.006.patch
>
>
> TestRMNMInfo fails intermittently. Below is trace for the failure
> {noformat}
> testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed: 0.28
> sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but
> was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4686) MiniYARNCluster.start() returns before cluster is completely started
[
https://issues.apache.org/jira/browse/YARN-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated YARN-4686:
--
Attachment: (was: YARN-4686-branch-2.7.007.patch)
> MiniYARNCluster.start() returns before cluster is completely started
>
>
> Key: YARN-4686
> URL: https://issues.apache.org/jira/browse/YARN-4686
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Eric Badger
> Attachments: MAPREDUCE-6507.001.patch,
> YARN-4686-branch-2.7.006.patch, YARN-4686.001.patch, YARN-4686.002.patch,
> YARN-4686.003.patch, YARN-4686.004.patch, YARN-4686.005.patch,
> YARN-4686.006.patch
>
>
> TestRMNMInfo fails intermittently. Below is trace for the failure
> {noformat}
> testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed: 0.28
> sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but
> was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4686) MiniYARNCluster.start() returns before cluster is completely started
[
https://issues.apache.org/jira/browse/YARN-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204359#comment-15204359
]
Hadoop QA commented on YARN-4686:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s {color}
| {color:red} YARN-4686 does not apply to branch-2.7. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12794518/YARN-4686-branch-2.7.007.patch
|
| JIRA Issue | YARN-4686 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/10834/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> MiniYARNCluster.start() returns before cluster is completely started
>
>
> Key: YARN-4686
> URL: https://issues.apache.org/jira/browse/YARN-4686
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Eric Badger
> Attachments: MAPREDUCE-6507.001.patch,
> YARN-4686-branch-2.7.006.patch, YARN-4686-branch-2.7.007.patch,
> YARN-4686.001.patch, YARN-4686.002.patch, YARN-4686.003.patch,
> YARN-4686.004.patch, YARN-4686.005.patch, YARN-4686.006.patch
>
>
> TestRMNMInfo fails intermittently. Below is trace for the failure
> {noformat}
> testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed: 0.28
> sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but
> was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4686) MiniYARNCluster.start() returns before cluster is completely started
[
https://issues.apache.org/jira/browse/YARN-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated YARN-4686:
--
Attachment: YARN-4686-branch-2.7.007.patch
Attaching branch-2.7 patch after rebase. Must not have done that initially.
> MiniYARNCluster.start() returns before cluster is completely started
>
>
> Key: YARN-4686
> URL: https://issues.apache.org/jira/browse/YARN-4686
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Eric Badger
> Attachments: MAPREDUCE-6507.001.patch,
> YARN-4686-branch-2.7.006.patch, YARN-4686-branch-2.7.007.patch,
> YARN-4686.001.patch, YARN-4686.002.patch, YARN-4686.003.patch,
> YARN-4686.004.patch, YARN-4686.005.patch, YARN-4686.006.patch
>
>
> TestRMNMInfo fails intermittently. Below is trace for the failure
> {noformat}
> testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed: 0.28
> sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but
> was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4842) yarn logs command should not require the appOwner argument
[ https://issues.apache.org/jira/browse/YARN-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ram Venkatesh updated YARN-4842: Attachment: YARN-4842.2.patch Fixed failing unit tests, added a test for this change > yarn logs command should not require the appOwner argument > -- > > Key: YARN-4842 > URL: https://issues.apache.org/jira/browse/YARN-4842 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ram Venkatesh >Assignee: Ram Venkatesh > Attachments: YARN-4842.1.patch, YARN-4842.2.patch > > > The yarn logs command is among the most common ways to troubleshoot yarn app > failures, especially by an admin. > Currently if you run the command as a user different from the job owner, the > command will fail with a subtle message that it could not find the app under > the running user's name. This can be confusing especially to new admins. > We can figure out the job owner from the app report returned by the RM or the > AHS, or, by looking for the app directory using a glob pattern, so in most > cases this error can be avoided. > Question - are there scenarios where users will still need to specify the > -appOwner option? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204195#comment-15204195 ] Yi Zhou commented on YARN-4847: --- [~Naganarasimha] Thanks a lot for this work ! > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204191#comment-15204191 ] Yi Zhou commented on YARN-796: -- BTW, i am confused that there is no 'Node Labels' in left navigation on 'ResourceManager' UI even after enabling the node label feature..It is not yet supported in 2.6 ? > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-998) Persistent resource change during NM/RM restart
[ https://issues.apache.org/jira/browse/YARN-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-998: Attachment: YARN-998-v2.1.patch Update the patch to address checkstyle issue. > Persistent resource change during NM/RM restart > --- > > Key: YARN-998 > URL: https://issues.apache.org/jira/browse/YARN-998 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-998-sample.patch, YARN-998-v1.patch, > YARN-998-v2.1.patch, YARN-998-v2.patch > > > When NM is restarted by plan or from a failure, previous dynamic resource > setting should be kept for consistency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204184#comment-15204184 ] Yi Zhou commented on YARN-796: -- I configured below in capacity-scheduler.xml and try to restart RM but it always got failure with "Illegal capacity of 1.0 for children of queue root for label=y". I think the below setting for CS seem be OK and not very sure why it happened. Could you please help me to have a look for this ? Thanks ! yarn.scheduler.capacity.root.queues e,m,s yarn.scheduler.capacity.root.e.capacity 33 yarn.scheduler.capacity.root.m.capacity 34 yarn.scheduler.capacity.root.s.capacity 33 yarn.scheduler.capacity.root.e.accessible-node-labels y yarn.scheduler.capacity.root.m.accessible-node-labels y yarn.scheduler.capacity.root.e.accessible-node-labels.y.capacity 50 yarn.scheduler.capacity.root.m.accessible-node-labels.y.capacity 50 yarn.scheduler.capacity.root.e.default-node-label-expression y Caused by: java.io.IOException: Failed to re-init queues at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:358) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:363) ... 10 more Caused by: java.lang.IllegalArgumentException: Illegal capacity of 1.0 for children of queue root for label=y at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setChildQueues(ParentQueue.java:184) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:594) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:477) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:355) ... 11 more > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4843) [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to int64
[ https://issues.apache.org/jira/browse/YARN-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204177#comment-15204177 ] Junping Du commented on YARN-4843: -- Thanks [~leftnoteasy] for filing a JIRA to address this issue. I think we may have to add a new field instead of change directly on existing field or we will have incompatible issues. > [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to > int64 > - > > Key: YARN-4843 > URL: https://issues.apache.org/jira/browse/YARN-4843 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Reporter: Wangda Tan > > This JIRA is to track all int32 usages in YARN's ProtocolBuffer APIs that we > possibly need to update to int64. > One example is resource API. We use int32 for memory now, if a cluster has > 10k nodes, each node has 210G memory, we will get a negative total cluster > memory. > We may have other fields may need to upgrade from int32 to int64. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204117#comment-15204117 ] Yi Zhou commented on YARN-796: -- Hi, Successfully assigned the node label as per your instruction in 2.6 ! I will continue to make progress.. sudo -u yarn yarn rmadmin -replaceLabelsOnNode "bb-node4,y" 16/03/21 20:38:03 INFO client.RMProxy: Connecting to ResourceManager at bb-node1/192.168.30.1:8033 [root@bb-node1 ~]# sudo -u yarn yarn node -status bb-node4:8041 16/03/21 20:39:09 INFO client.RMProxy: Connecting to ResourceManager at bb-node1/192.168.30.1:8032 Node Report : Node-Id : bb-node4:8041 Rack : /default Node-State : RUNNING Node-Http-Address : bb-node4:8042 Last-Health-Update : Mon 21/Mar/16 08:39:06:548CST Health-Report : Containers : 0 Memory-Used : 0MB Memory-Capacity : 65536MB CPU-Used : 0 vcores CPU-Capacity : 40 vcores Node-Labels : y > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4517) [YARN-3368] Add nodes page
[ https://issues.apache.org/jira/browse/YARN-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204115#comment-15204115 ] Varun Saxena commented on YARN-4517: Filed YARN-4848 to address above comments so that it can unblock UI merge to trunk, for YARN-2928. Do we need AM pages too for merge to trunk ? I guess we can merge regardless of it so that UI related work on YARN-2928 can speedup. Do let me know. > [YARN-3368] Add nodes page > -- > > Key: YARN-4517 > URL: https://issues.apache.org/jira/browse/YARN-4517 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Wangda Tan >Assignee: Varun Saxena > Labels: webui > Attachments: (21-Feb-2016)yarn-ui-screenshots.zip, > Screenshot_after_4709.png, Screenshot_after_4709_1.png, > YARN-4517-YARN-3368.01.patch, YARN-4517-YARN-3368.02.patch > > > We need nodes page added to next generation web UI, similar to existing > RM/nodes page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4848) [YARN-3368] Unify app/container pages and other miscellaneous issues
[ https://issues.apache.org/jira/browse/YARN-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204102#comment-15204102 ] Varun Saxena commented on YARN-4848: In addition to this, we need to resolve some of the issues mentioned in the link https://issues.apache.org/jira/browse/YARN-4517?focusedCommentId=15203840&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15203840 > [YARN-3368] Unify app/container pages and other miscellaneous issues > > > Key: YARN-4848 > URL: https://issues.apache.org/jira/browse/YARN-4848 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Reporter: Varun Saxena >Assignee: Varun Saxena > > We need to consolidate app and container views. Currently apps and container > views from NM page do not land up on a unified app/container page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4848) [YARN-3368] Unify app/container pages and other miscellaneous issues
Varun Saxena created YARN-4848: -- Summary: [YARN-3368] Unify app/container pages and other miscellaneous issues Key: YARN-4848 URL: https://issues.apache.org/jira/browse/YARN-4848 Project: Hadoop YARN Issue Type: Sub-task Components: webapp Reporter: Varun Saxena Assignee: Varun Saxena We need to consolidate app and container views. Currently apps and container views from NM page do not land up on a unified app/container page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204100#comment-15204100 ] Yi Zhou commented on YARN-796: -- OK i will also try it as per your suggestion in my cluster. Thanks again guys ! > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204085#comment-15204085 ] Naganarasimha G R commented on YARN-796: yes "=" is not supported... again missed it ! I think document updation is very important have raised "YARN-4847" for the same > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-4847: Summary: Add documentation for the Node Label features supported in 2.6 (was: Add documentation to the Node Label features supported in 2.6 ) > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4847) Add documentation to the features supported in 2.6
Naganarasimha G R created YARN-4847: --- Summary: Add documentation to the features supported in 2.6 Key: YARN-4847 URL: https://issues.apache.org/jira/browse/YARN-4847 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.6.4 Reporter: Naganarasimha G R Assignee: Naganarasimha G R We constantly face issue with what are the node label supported features in 2.6 and general commands to use it. So it would be better to have documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4847) Add documentation to the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-4847: Summary: Add documentation to the Node Label features supported in 2.6 (was: Add documentation to the features supported in 2.6 ) > Add documentation to the Node Label features supported in 2.6 > -- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[
https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204080#comment-15204080
]
Sunil G commented on YARN-796:
--
Hi
Cud u pls modify {{"=" to ","}} in Step 5 and try once.
{noformat}
5) Assign Node Labels to Cluster Node
sudo -u yarn yarn rmadmin -replaceLabelsOnNode "bb-node3,y"
{noformat}
I am not very sure whether we support "=" in 2.6 now. May be [~Naganarasimha]
can confirm.
> Allow for (admin) labels on nodes and resource-requests
> ---
>
> Key: YARN-796
> URL: https://issues.apache.org/jira/browse/YARN-796
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.4.1
>Reporter: Arun C Murthy
>Assignee: Wangda Tan
> Attachments: LabelBasedScheduling.pdf,
> Node-labels-Requirements-Design-doc-V1.pdf,
> Node-labels-Requirements-Design-doc-V2.pdf,
> Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf,
> YARN-796.node-label.consolidate.1.patch,
> YARN-796.node-label.consolidate.10.patch,
> YARN-796.node-label.consolidate.11.patch,
> YARN-796.node-label.consolidate.12.patch,
> YARN-796.node-label.consolidate.13.patch,
> YARN-796.node-label.consolidate.14.patch,
> YARN-796.node-label.consolidate.2.patch,
> YARN-796.node-label.consolidate.3.patch,
> YARN-796.node-label.consolidate.4.patch,
> YARN-796.node-label.consolidate.5.patch,
> YARN-796.node-label.consolidate.6.patch,
> YARN-796.node-label.consolidate.7.patch,
> YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1,
> YARN-796.patch, YARN-796.patch4
>
>
> It will be useful for admins to specify labels for nodes. Examples of labels
> are OS, processor architecture etc.
> We should expose these labels and allow applications to specify labels on
> resource-requests.
> Obviously we need to support admin operations on adding/removing node labels.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204074#comment-15204074 ] Naganarasimha G R commented on YARN-796: Hi [~jameszhouyi], I have always tried in local folder, let me test in 2.6 as per your steps later will update you. > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204071#comment-15204071 ] Yi Zhou commented on YARN-796: -- Hi, I found there can't take effect after assigning a created node label to a node in 2.6.0. You can find that there is EMPTY in ''Node-Labels :" row by 'yarn node -status XX'. Please kindly see below steps for details. If there is a workaround for this issue ? Thanks advance ! step: 1) Create a Label Directory in HDFS sudo -u hdfs hadoop fs -mkdir -p /yarn/node-labels sudo -u hdfs hadoop fs -chown -R yarn:yarn /yarn sudo -u hdfs hadoop fs -chmod -R 700 /yarn sudo -u hdfs hadoop fs -ls /yarn 2) Configure YARN for Node Labels yarn.node-labels.enabled true yarn.node-labels.fs-store.root-dir hdfs://bb-node1:8020/yarn/node-labels/ 3) Restart RM service 4) Add Node Labels sudo -u yarn yarn rmadmin -addToClusterNodeLabels "y" 5) Assign Node Labels to Cluster Node sudo -u yarn yarn rmadmin -replaceLabelsOnNode "bb-node3=y" 6) Check the Node label sudo -u yarn yarn node -status bb-node3:8041 16/03/21 19:37:12 INFO client.RMProxy: Connecting to ResourceManager at bb-node1/192.168.30.1:8032 Node Report : Node-Id : bb-node3:8041 Rack : /default Node-State : RUNNING Node-Http-Address : bb-node3:8042 Last-Health-Update : Mon 21/Mar/16 07:36:42:522CST Health-Report : Containers : 0 Memory-Used : 0MB Memory-Capacity : 65536MB CPU-Used : 0 vcores CPU-Capacity : 40 vcores Node-Labels : > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4517) [YARN-3368] Add nodes page
[ https://issues.apache.org/jira/browse/YARN-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203914#comment-15203914 ] Varun Saxena commented on YARN-4517: [~leftnoteasy], thanks for the review. bq. Not sure why url looks like following: http://localhost:4200/yarnNode/localhost:61236/localhost:8042, I can understand last part is nodeId.toString, but why is localhost:61236 added to URL? NodeId is required to query node info from RM. It has been passed around everywhere because we need to create link for left hand side menu on every page. bq. Cannot view container logs when log aggregation is disabled With aggregation disabled ? Are you testing with latest trunk code ? We had issues before YARN-4709 went in. bq. hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/app/controllers/application.js typically we shouldn't put html directly to controller, ideally we should move as much as possible html code into templates. If we cannot move all of them into templates, component/helper should be fine with html code. Ok. Moved this to controller to get hold of currentPath. Will move this code to a helper or component. bq. Format issue: pie chart - Resource - Memory (in MB) cannot be fully rendered from my browser Ok. Will check. Even with full(not minimized) browser window ? > [YARN-3368] Add nodes page > -- > > Key: YARN-4517 > URL: https://issues.apache.org/jira/browse/YARN-4517 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Wangda Tan >Assignee: Varun Saxena > Labels: webui > Attachments: (21-Feb-2016)yarn-ui-screenshots.zip, > Screenshot_after_4709.png, Screenshot_after_4709_1.png, > YARN-4517-YARN-3368.01.patch, YARN-4517-YARN-3368.02.patch > > > We need nodes page added to next generation web UI, similar to existing > RM/nodes page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4842) yarn logs command should not require the appOwner argument
[
https://issues.apache.org/jira/browse/YARN-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203893#comment-15203893
]
Hadoop QA commented on YARN-4842:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 58s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 3 new +
50 unchanged - 1 fixed = 53 total (was 51) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 9s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 26s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_74.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 21s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 40s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:g
[jira] [Updated] (YARN-4846) Random failures for TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers
[
https://issues.apache.org/jira/browse/YARN-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-4846:
---
Summary: Random failures for
TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers
(was: Randome failures for
TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers)
> Random failures for
> TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers
>
>
> Key: YARN-4846
> URL: https://issues.apache.org/jira/browse/YARN-4846
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerPreemption.testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers(TestCapacitySchedulerPreemption.java:473)
> {noformat}
> https://builds.apache.org/job/PreCommit-YARN-Build/10826/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacitySchedulerPreemption/testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4846) Randome failures for TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers
[
https://issues.apache.org/jira/browse/YARN-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-4846:
---
Issue Type: Sub-task (was: Bug)
Parent: YARN-4478
> Randome failures for
> TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers
> -
>
> Key: YARN-4846
> URL: https://issues.apache.org/jira/browse/YARN-4846
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerPreemption.testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers(TestCapacitySchedulerPreemption.java:473)
> {noformat}
> https://builds.apache.org/job/PreCommit-YARN-Build/10826/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacitySchedulerPreemption/testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4846) Randome failures for TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers
Bibin A Chundatt created YARN-4846:
--
Summary: Randome failures for
TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers
Key: YARN-4846
URL: https://issues.apache.org/jira/browse/YARN-4846
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
{noformat}
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerPreemption.testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers(TestCapacitySchedulerPreemption.java:473)
{noformat}
https://builds.apache.org/job/PreCommit-YARN-Build/10826/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacitySchedulerPreemption/testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4607) AppAttempt page TotalOutstandingResource Requests table support pagination
[ https://issues.apache.org/jira/browse/YARN-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-4607: --- Attachment: Request with Pagination.jpg Request_without_pagination.jpg 0002-YARN-4607.patch Attaching updated patch and also screen shot for attempt page. > AppAttempt page TotalOutstandingResource Requests table support pagination > -- > > Key: YARN-4607 > URL: https://issues.apache.org/jira/browse/YARN-4607 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: 0001-YARN-4607.patch, 0002-YARN-4607.patch, Request with > Pagination.jpg, Request_without_pagination.jpg > > > Simulate cluster with 10 racks with 100 nodes using sls and of we check the > table for Total Outstanding Resource Requests will consume complete page. > Good to support pagination for the table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4517) [YARN-3368] Add nodes page
[
https://issues.apache.org/jira/browse/YARN-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203840#comment-15203840
]
Wangda Tan commented on YARN-4517:
--
Thanks [~varun_saxena] working this JIRA and [~gtCarrera]/[~sunilg] for
discussions. In general it looks good.
A couple of comments,
- Not sure why url looks like following:
{{http://localhost:4200/yarnNode/localhost:61236/localhost:8042}}, I can
understand last part is nodeId.toString, but why is {{localhost:61236}} added
to URL?
- Cannot view container logs when log aggregation is disabled
- Format issue: pie chart - {{Resource - Memory (in MB)}} cannot be fully
rendered from my browser
-
{{hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/app/controllers/application.js}}
typically we shouldn't put html directly to controller, ideally we should move
as much as possible html code into templates. If we cannot move all of them
into templates, component/helper should be fine with html code.
- As mentioned by [~gtCarrera], we need a unified container/application page.
Since we're trying to merge code back to trunk soon to unblock YARN-2928, I
think it will be fine to address above comments in a separated JIRA.
To make sure we won't forget what we discussed above, could you file a ticket
to fix above items?
> [YARN-3368] Add nodes page
> --
>
> Key: YARN-4517
> URL: https://issues.apache.org/jira/browse/YARN-4517
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Wangda Tan
>Assignee: Varun Saxena
> Labels: webui
> Attachments: (21-Feb-2016)yarn-ui-screenshots.zip,
> Screenshot_after_4709.png, Screenshot_after_4709_1.png,
> YARN-4517-YARN-3368.01.patch, YARN-4517-YARN-3368.02.patch
>
>
> We need nodes page added to next generation web UI, similar to existing
> RM/nodes page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
