date:20161215


[ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753519#comment-15753519
 ] 

zhangyubiao commented on YARN-5995:
---

Fix asflicense and FindBugs, YARN-5995.0002.patch is commit.

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Attachments: YARN-5995.0001.patch, YARN-5995.0002.patch, 
> YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


 [ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangyubiao updated YARN-5995:
--
Attachment: YARN-5995.0002.patch

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Attachments: YARN-5995.0001.patch, YARN-5995.0002.patch, 
> YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3866) AM-RM protocol changes to support container resizing


[ 
https://issues.apache.org/jira/browse/YARN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753429#comment-15753429
 ] 

Hadoop QA commented on YARN-3866:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 
47s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m 
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 7s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
29s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
19s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
35s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
5s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
58s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
3s{color} | {color:green} branch-2.8 passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
20s{color} | {color:red} hadoop-yarn-server-resourcemanager in branch-2.8 
failed with JDK v1.8.0_111. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
31s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  1m 48s{color} 
| {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.8.0_111 with JDK v1.8.0_111 
generated 18 new + 39 unchanged - 0 fixed = 57 total (was 39) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
9s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  2m  
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  2m  9s{color} 
| {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.7.0_121 with JDK v1.7.0_121 
generated 20 new + 48 unchanged - 0 fixed = 68 total (was 48) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 33s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 32 new + 102 unchanged - 0 fixed = 134 total (was 102) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
20s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed 
with JDK v1.8.0_111. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
25s{color} | {color:green} hadoop-yarn-api in the patch passed with JDK 
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
24s{color} | {color:green} hadoop-yarn-common in the patch passed with JDK

[jira] [Commented] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


[ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753412#comment-15753412
 ] 

Hadoop QA commented on YARN-5995:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 
53s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
 0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 19s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 27 new + 45 unchanged - 2 fixed = 72 total (was 47) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m  
6s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  6m 40s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
17s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 50s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
|  |  Write to static field 
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.rmStateStoreOpDurations
 from instance method new 
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore()  At 
RMStateStore.java:from instance method new 
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore()  At 
RMStateStore.java:[line 642] |
|  |  Static initializer for 
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStoreOpDurations 
creates instance before all static final fields assigned  At 
RMStateStoreOpDurations.java:instance before all static final fields assigned  
At RMStateStoreOpDurations.java:[line 66] |
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling |
|   | hadoop.yarn.server.resourcemanager.TestRMHA |
|   | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
|   | hadoop.yarn.server.resourcemanager.TestTokenClientRMService |
|   | hadoop.yarn.server.resourcemanager.rmapp.TestApplicationLifetimeMonitor |
|   | 
hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections

[jira] [Commented] (YARN-5967) Fix slider core module findbugs warnings


[ 
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753382#comment-15753382
 ] 

Jian He commented on YARN-5967:
---

- For any unused code that's throwing warnings, simply removed it
- Suppress certain warnings which are ok.

> Fix slider core module findbugs warnings 
> -
>
> Key: YARN-5967
> URL: https://issues.apache.org/jira/browse/YARN-5967
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5967-yarn-native-services.01.patch, 
> YARN-5967-yarn-native-services.02.patch, 
> YARN-5967-yarn-native-services.02.patch, 
> YARN-5967-yarn-native-services.03.patch, 
> YARN-5967-yarn-native-services.04.patch, 
> YARN-5967-yarn-native-services.04.patch, 
> YARN-5967-yarn-native-services.05.patch, 
> YARN-5967-yarn-native-services.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5646) Documentation for scheduling of OPPORTUNISTIC containers


 [ 
https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos updated YARN-5646:
-
Attachment: YARN-5646.004.patch

> Documentation for scheduling of OPPORTUNISTIC containers
> 
>
> Key: YARN-5646
> URL: https://issues.apache.org/jira/browse/YARN-5646
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
>Priority: Blocker
> Attachments: YARN-5646.001.patch, YARN-5646.002.patch, 
> YARN-5646.003.patch, YARN-5646.004.patch
>
>
> This is for adding documentation regarding the scheduling of OPPORTUNISTIC 
> containers.
> It includes both the centralized (YARN-5220) and the distributed (YARN-2877) 
> scheduling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5740) Add a new field in Slider status output - lifetime (remaining)


 [ 
https://issues.apache.org/jira/browse/YARN-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5740:
--
Attachment: YARN-5740-yarn-native-services.03.patch

> Add a new field in Slider status output - lifetime (remaining)
> --
>
> Key: YARN-5740
> URL: https://issues.apache.org/jira/browse/YARN-5740
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Jian He
> Fix For: yarn-native-services
>
> Attachments: YARN-5740-yarn-native-services.01.patch, 
> YARN-5740-yarn-native-services.02.patch, 
> YARN-5740-yarn-native-services.03.patch
>
>
> With YARN-5735, REST service is now setting lifetime to application during 
> submission (YARN-4205 exposed application lifetime support). Now Slider 
> status needs to expose this field so that the REST service can return it in 
> its GET response. Note, the lifetime value that GET response intends to 
> return is the remaining lifetime of the application. 
> There is an ongoing discussion in YARN-4206, that the lifetime value returned 
> in Application Report will be the remaining lifetime (at the time of 
> request). So until it is finalized, the lifetime value might go through 
> different connotations. But as long as we have the lifetime field in the 
> status output, it will be a good start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


[ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753327#comment-15753327
 ] 

zhangyubiao commented on YARN-5995:
---

Thanks [~sunilg], I update YARN-5995.0001.patch 

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Attachments: YARN-5995.0001.patch, YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


 [ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangyubiao updated YARN-5995:
--
Attachment: YARN-5995.0001.patch

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Attachments: YARN-5995.0001.patch, YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


 [ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangyubiao updated YARN-5995:
--
Attachment: (was: YARN-5995-v1.patch)

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Attachments: YARN-5995.0001.patch, YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance

2016-12-15 Thread Sunil G (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753316#comment-15753316
 ] 

Sunil G commented on YARN-5995:
---

Hi [~piaoyu zhang]
one general comment. Patch name format can be like {{YARN-5995.000x.patch}} 
where 'x' indicates revisions like 1,2 etc. It will be easier to follow such a 
format which is used generally. 

I ll take a detailed look on patch and will share some comments.

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Attachments: YARN-5995-v1.patch, YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


 [ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangyubiao updated YARN-5995:
--
Fix Version/s: (was: 2.7.4)

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Attachments: YARN-5995-v1.patch, YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


 [ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangyubiao updated YARN-5995:
--
Target Version/s: 3.0.0-alpha2  (was: 3.0.0-beta1)
   Fix Version/s: (was: 3.0.0-beta1)

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Fix For: 2.7.4
>
> Attachments: YARN-5995-v1.patch, YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5646) Documentation for scheduling of OPPORTUNISTIC containers


 [ 
https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos updated YARN-5646:
-
Attachment: YARN-5646.003.patch

Also updating yarn-default.xml with new variable names.

> Documentation for scheduling of OPPORTUNISTIC containers
> 
>
> Key: YARN-5646
> URL: https://issues.apache.org/jira/browse/YARN-5646
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
>Priority: Blocker
> Attachments: YARN-5646.001.patch, YARN-5646.002.patch, 
> YARN-5646.003.patch
>
>
> This is for adding documentation regarding the scheduling of OPPORTUNISTIC 
> containers.
> It includes both the centralized (YARN-5220) and the distributed (YARN-2877) 
> scheduling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5646) Documentation for scheduling of OPPORTUNISTIC containers


 [ 
https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos updated YARN-5646:
-
Attachment: YARN-5646.002.patch

Adding updated documentation.
Thanks again for all the feedback, [~templedf], [~curino], [~kasha], [~asuresh].
I have addressed almost all your comments. I also reorganized some sections, 
adding an initial "Quick Guide" that can be used to quickly enable and tried 
out the new feature (without requiring to go through all the system details).

> Documentation for scheduling of OPPORTUNISTIC containers
> 
>
> Key: YARN-5646
> URL: https://issues.apache.org/jira/browse/YARN-5646
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
>Priority: Blocker
> Attachments: YARN-5646.001.patch, YARN-5646.002.patch
>
>
> This is for adding documentation regarding the scheduling of OPPORTUNISTIC 
> containers.
> It includes both the centralized (YARN-5220) and the distributed (YARN-2877) 
> scheduling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5740) Add a new field in Slider status output - lifetime (remaining)


[ 
https://issues.apache.org/jira/browse/YARN-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753152#comment-15753152
 ] 

Hadoop QA commented on YARN-5740:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 8s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
37s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m  
9s{color} | {color:red} hadoop-yarn-services-api in yarn-native-services 
failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
28s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
51s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
 in yarn-native-services has 262 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
26s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api
 in yarn-native-services has 1 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
26s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services 
failed. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 21s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications: The patch generated 
19 new + 405 unchanged - 3 fixed = 424 total (was 408) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m  
8s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
24s{color} | {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
39s{color} | {color:green} hadoop-yarn-slider-core in the patch passed. {color} 
|
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
16s{color} | {color:green} hadoop-yarn-services-api in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 54s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5740 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843520/YARN-5740-yarn-native-services.02.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux a6d80a673049 3.13.0-103-generic

[jira] [Commented] (YARN-5967) Fix slider core module findbugs warnings


[ 
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753124#comment-15753124
 ] 

Hadoop QA commented on YARN-5967:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  4m 
59s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
57s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
28s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
4s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
32s{color} | {color:green} yarn-native-services passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m  
2s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
 in yarn-native-services has 262 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
28s{color} | {color:red} hadoop-yarn-slider in yarn-native-services failed. 
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
30s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services 
failed. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 27s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider: 
The patch generated 31 new + 904 unchanged - 61 fixed = 935 total (was 965) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider 
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
4s{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
 generated 0 new + 6 unchanged - 256 fixed = 6 total (was 262) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
27s{color} | {color:red} hadoop-yarn-slider in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
26s{color} | {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
43s{color} | {color:green} hadoop-yarn-slider in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
41s{color} | {color:green} hadoop-yarn-slider-core in the patch passed. {color} 
|
|

[jira] [Commented] (YARN-5740) Add a new field in Slider status output - lifetime (remaining)


[ 
https://issues.apache.org/jira/browse/YARN-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753103#comment-15753103
 ] 

Jian He commented on YARN-5740:
---

thanks for the review, fixed the issues except one, which I think no need to 
fix because all other variables do the same.
{{./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/java/org/apache/slider/common/params/ActionStatusArgs.java:40:
  public boolean lifetime;:18: Variable 'lifetime' must be private and have 
accessor methods.}}

> Add a new field in Slider status output - lifetime (remaining)
> --
>
> Key: YARN-5740
> URL: https://issues.apache.org/jira/browse/YARN-5740
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Jian He
> Fix For: yarn-native-services
>
> Attachments: YARN-5740-yarn-native-services.01.patch, 
> YARN-5740-yarn-native-services.02.patch
>
>
> With YARN-5735, REST service is now setting lifetime to application during 
> submission (YARN-4205 exposed application lifetime support). Now Slider 
> status needs to expose this field so that the REST service can return it in 
> its GET response. Note, the lifetime value that GET response intends to 
> return is the remaining lifetime of the application. 
> There is an ongoing discussion in YARN-4206, that the lifetime value returned 
> in Application Report will be the remaining lifetime (at the time of 
> request). So until it is finalized, the lifetime value might go through 
> different connotations. But as long as we have the lifetime field in the 
> status output, it will be a good start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5740) Add a new field in Slider status output - lifetime (remaining)


 [ 
https://issues.apache.org/jira/browse/YARN-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5740:
--
Attachment: (was: YARN-5740-yarn-native-services.02.patch)

> Add a new field in Slider status output - lifetime (remaining)
> --
>
> Key: YARN-5740
> URL: https://issues.apache.org/jira/browse/YARN-5740
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Jian He
> Fix For: yarn-native-services
>
> Attachments: YARN-5740-yarn-native-services.01.patch, 
> YARN-5740-yarn-native-services.02.patch
>
>
> With YARN-5735, REST service is now setting lifetime to application during 
> submission (YARN-4205 exposed application lifetime support). Now Slider 
> status needs to expose this field so that the REST service can return it in 
> its GET response. Note, the lifetime value that GET response intends to 
> return is the remaining lifetime of the application. 
> There is an ongoing discussion in YARN-4206, that the lifetime value returned 
> in Application Report will be the remaining lifetime (at the time of 
> request). So until it is finalized, the lifetime value might go through 
> different connotations. But as long as we have the lifetime field in the 
> status output, it will be a good start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5740) Add a new field in Slider status output - lifetime (remaining)


 [ 
https://issues.apache.org/jira/browse/YARN-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5740:
--
Attachment: YARN-5740-yarn-native-services.02.patch

> Add a new field in Slider status output - lifetime (remaining)
> --
>
> Key: YARN-5740
> URL: https://issues.apache.org/jira/browse/YARN-5740
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Jian He
> Fix For: yarn-native-services
>
> Attachments: YARN-5740-yarn-native-services.01.patch, 
> YARN-5740-yarn-native-services.02.patch
>
>
> With YARN-5735, REST service is now setting lifetime to application during 
> submission (YARN-4205 exposed application lifetime support). Now Slider 
> status needs to expose this field so that the REST service can return it in 
> its GET response. Note, the lifetime value that GET response intends to 
> return is the remaining lifetime of the application. 
> There is an ongoing discussion in YARN-4206, that the lifetime value returned 
> in Application Report will be the remaining lifetime (at the time of 
> request). So until it is finalized, the lifetime value might go through 
> different connotations. But as long as we have the lifetime field in the 
> status output, it will be a good start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5740) Add a new field in Slider status output - lifetime (remaining)


 [ 
https://issues.apache.org/jira/browse/YARN-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5740:
--
Attachment: YARN-5740-yarn-native-services.02.patch

> Add a new field in Slider status output - lifetime (remaining)
> --
>
> Key: YARN-5740
> URL: https://issues.apache.org/jira/browse/YARN-5740
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Jian He
> Fix For: yarn-native-services
>
> Attachments: YARN-5740-yarn-native-services.01.patch, 
> YARN-5740-yarn-native-services.02.patch
>
>
> With YARN-5735, REST service is now setting lifetime to application during 
> submission (YARN-4205 exposed application lifetime support). Now Slider 
> status needs to expose this field so that the REST service can return it in 
> its GET response. Note, the lifetime value that GET response intends to 
> return is the remaining lifetime of the application. 
> There is an ongoing discussion in YARN-4206, that the lifetime value returned 
> in Application Report will be the remaining lifetime (at the time of 
> request). So until it is finalized, the lifetime value might go through 
> different connotations. But as long as we have the lifetime field in the 
> status output, it will be a good start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5756) Add state-machine implementation for queues


[ 
https://issues.apache.org/jira/browse/YARN-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753079#comment-15753079
 ] 

Wangda Tan commented on YARN-5756:
--

Thanks [~xgong],

Generally looks good, some minor comments:

1) AbstractCSQueue#initializeQueueState:
- initializeQueueState: is it better to rename current to previous?
- And can we make the initializeQueueState accept 3 parameters: 
previous/configured/parent, with this we can do a more easier and comprehensive 
UT to make sure it won't break in the future.

2) Leaf(Parent)Queue#haveActiveApps can move to abstractCSQueue, or just 
completely remove the method since we can use getNumApplications instead.

3) AbstractCSQueue#updateQueueState, writeLock is not necessary for volatile. 
{{queueStateWriteLock}} can be directly removed.

4) QueueStateManager is not used by anybody now? Is it better to move to a 
separated patch? Which JIRA you plan to use it?

> Add state-machine implementation for queues
> ---
>
> Key: YARN-5756
> URL: https://issues.apache.org/jira/browse/YARN-5756
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-5756.1.patch, YARN-5756.2.patch, YARN-5756.3.patch, 
> YARN-5756.4.patch, YARN-5756.5.patch, YARN-5756.6.patch, YARN-5756.6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5967) Fix slider core module findbugs warnings


 [ 
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5967:
--
Attachment: YARN-5967-yarn-native-services.06.patch

> Fix slider core module findbugs warnings 
> -
>
> Key: YARN-5967
> URL: https://issues.apache.org/jira/browse/YARN-5967
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5967-yarn-native-services.01.patch, 
> YARN-5967-yarn-native-services.02.patch, 
> YARN-5967-yarn-native-services.02.patch, 
> YARN-5967-yarn-native-services.03.patch, 
> YARN-5967-yarn-native-services.04.patch, 
> YARN-5967-yarn-native-services.04.patch, 
> YARN-5967-yarn-native-services.05.patch, 
> YARN-5967-yarn-native-services.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6006) Log aggregation causes nodemanager OOM


 [ 
https://issues.apache.org/jira/browse/YARN-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shen Yinjie updated YARN-6006:
--
Attachment: (was: YARN-6006_1.patch)

> Log aggregation causes nodemanager OOM
> --
>
> Key: YARN-6006
> URL: https://issues.apache.org/jira/browse/YARN-6006
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Shen Yinjie
> Attachments: e.png
>
>
> log aggregation is enabled, nodemanager died with oom exception. exception as 
> sreenshot. and NM_heapsize is 4096MB



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5987) NM configured command to collect heap dump of preempted container


[ 
https://issues.apache.org/jira/browse/YARN-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753007#comment-15753007
 ] 

Hadoop QA commented on YARN-5987:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
21s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 441 unchanged - 52 fixed = 441 total (was 493) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
34s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
30s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 
55s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
34s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 39s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5987 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843500/YARN-5987.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 3115b120c36b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / ac0d402 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14336/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api

[jira] [Commented] (YARN-3955) Support for priority ACLs in CapacityScheduler


[ 
https://issues.apache.org/jira/browse/YARN-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752985#comment-15752985
 ] 

Wangda Tan commented on YARN-3955:
--

Thanks [~sunilg]. 

Besides UT failures, findbugs, etc. Some comments:

1) AppPriorityACLsManager
- {{private class PriorityACL}} add static
- Why put it to RM(Active)Context? We have checkAndGetApplicationPriority 
exposed by scheduler already. Is it possible to remove it from contexts? 
- setPrioirityACLsToStore, remove the store? It doesn't have logic of state 
store? And where is the logic to update state store?
- Common logic of checkAccess / getDefaultPriority can be merged.
- getDefaultPriority, I found PriorityAclGroup#getDefaultPriority is not 
accessed, so this looks like using max priority instead of default. Adding some 
UTs for this?

2) Changes to capacity-scheduler.xml
- Related to this JIRA? 

3) CapacityScheduler
- checkAndGetApplicationPriority: priorityFromContext -> something like 
priorityRequestedByApp
- not reuse "priorityRequestedByApp", and change the appPriority
- this.appPriorityACLManager.getDefaultPriority can be called before calling 
queueManager.getDefaultPriorityForQueue. And getDefaultPriorityForQueue can be 
called only if previous one is null.

4) CSQueueManager:
- move setPriorityAcls to setQueueAcls?

5) ClientRMService:
Instead of call methods of appPriorityAclManager (I prefer to remove it, see 
#1). Can we call scheduler#checkAndGetApplicationPriority to achieve the same 
goal? 

6) LeafQueue:
- getPriorityAcls, readLock is not necessary. Add a final to priorityAcls 
should be enough.
- remove unncessary import

7) Add comprehensive tests to PriorityAclConfiguration should be better.

8) RMAppManager:
- Changes of the {{if (isRecovery && YarnConfiguration.isAclEnabled(conf) && 
scheduler instanceof CapacityScheduler)}} may not be necessary?

> Support for priority ACLs in CapacityScheduler
> --
>
> Key: YARN-3955
> URL: https://issues.apache.org/jira/browse/YARN-3955
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacityscheduler
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: ApplicationPriority-ACL.pdf, 
> ApplicationPriority-ACLs-v2.pdf, YARN-3955.0001.patch, YARN-3955.0002.patch, 
> YARN-3955.0003.patch, YARN-3955.v0.patch, YARN-3955.v1.patch, 
> YARN-3955.wip1.patch
>
>
> Support will be added for User-level access permission to use different 
> application-priorities. This is to avoid situations where all users try 
> running max priority in the cluster and thus degrading the value of 
> priorities.
> Access Control Lists can be set per priority level within each queue. Below 
> is an example configuration that can be added in capacity scheduler 
> configuration
> file for each Queue level.
> yarn.scheduler.capacity.root...acl=user1,user2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5647) [Security] Collector and reader side changes for loading auth filters and principals

2016-12-15 Thread Sangjin Lee (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752980#comment-15752980
 ] 

Sangjin Lee commented on YARN-5647:
---

One question regarding the use of 
{{TimelineDelegationTokenSecretManagerService}}. This is somewhat related with 
an offline discussion we had during our weekly status call.

It appears that {{TimelineDelegationTokenSecretManagerService}} uses a (local) 
LevelDB to store tokens and manage renewals, etc. That will work fine with a 
single-instance-based v.1.x, but how would that work with Timeline Service v.2?

> [Security] Collector and reader side changes for loading auth filters and 
> principals
> 
>
> Key: YARN-5647
> URL: https://issues.apache.org/jira/browse/YARN-5647
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: oct16-hard
> Attachments: YARN-5647-YARN-5355.wip.002.patch, 
> YARN-5647-YARN-5355.wip.003.patch, YARN-5647-YARN-5355.wip.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5987) NM configured command to collect heap dump of preempted container


 [ 
https://issues.apache.org/jira/browse/YARN-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Miklos Szegedi updated YARN-5987:
-
Attachment: YARN-5987.001.patch

Fixing test issues.

> NM configured command to collect heap dump of preempted container
> -
>
> Key: YARN-5987
> URL: https://issues.apache.org/jira/browse/YARN-5987
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-5987.000.patch, YARN-5987.001.patch
>
>
> The node manager can kill a container, if it exceeds the assigned memory 
> limits. It would be nice to have a configuration entry to set up a command 
> that can collect additional debug information, if needed. The collected 
> information can be used for root cause analysis.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5600) Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application basis


[ 
https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752808#comment-15752808
 ] 

Hadoop QA commented on YARN-5600:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
 4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
34s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
59s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 555 unchanged - 21 fixed = 555 total (was 576) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
54s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 
42s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
32s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 33s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5600 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843484/YARN-5600.017.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux a70a8d73949d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 631f1da |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14334/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api

[jira] [Commented] (YARN-3866) AM-RM protocol changes to support container resizing


[ 
https://issues.apache.org/jira/browse/YARN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752785#comment-15752785
 ] 

Wangda Tan commented on YARN-3866:
--

Thanks [~andrew.wang] for the comment. We all aware the policy, and the latest 
changes should fix the incompat issue. I'm waiting for Jenkins result and we 
need to trigger another JACC run.

> AM-RM protocol changes to support container resizing
> 
>
> Key: YARN-3866
> URL: https://issues.apache.org/jira/browse/YARN-3866
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api
>Reporter: MENG DING
>Assignee: MENG DING
>Priority: Blocker
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: YARN-3866-YARN-1197.4.patch, 
> YARN-3866-branch-2.8.addendum.1.patch, YARN-3866-branch-2.8.addendum.1.patch, 
> YARN-3866-branch-2.8.addendum.2.patch, YARN-3866-branch-2.8.addendum.3.patch, 
> YARN-3866.1.patch, YARN-3866.2.patch, YARN-3866.3.patch
>
>
> YARN-1447 and YARN-1448 are outdated. 
> This ticket deals with AM-RM Protocol changes to support container resize 
> according to the latest design in YARN-1197.
> 1) Add increase/decrease requests in AllocateRequest
> 2) Get approved increase/decrease requests from RM in AllocateResponse
> 3) Add relevant test cases



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3866) AM-RM protocol changes to support container resizing


[ 
https://issues.apache.org/jira/browse/YARN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752768#comment-15752768
 ] 

Andrew Wang commented on YARN-3866:
---

Our compat guidelines available on the website, e.g.:

http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Compatibility.html

I think this is what applies here:

{quote}
Public-Stable APIs must be deprecated for at least one major release prior to 
their removal in a major release.
{quote}

Unless these APIs were deprecated in 2.1.0-beta, then we should wait to remove.

> AM-RM protocol changes to support container resizing
> 
>
> Key: YARN-3866
> URL: https://issues.apache.org/jira/browse/YARN-3866
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api
>Reporter: MENG DING
>Assignee: MENG DING
>Priority: Blocker
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: YARN-3866-YARN-1197.4.patch, 
> YARN-3866-branch-2.8.addendum.1.patch, YARN-3866-branch-2.8.addendum.1.patch, 
> YARN-3866-branch-2.8.addendum.2.patch, YARN-3866-branch-2.8.addendum.3.patch, 
> YARN-3866.1.patch, YARN-3866.2.patch, YARN-3866.3.patch
>
>
> YARN-1447 and YARN-1448 are outdated. 
> This ticket deals with AM-RM Protocol changes to support container resize 
> according to the latest design in YARN-1197.
> 1) Add increase/decrease requests in AllocateRequest
> 2) Get approved increase/decrease requests from RM in AllocateResponse
> 3) Add relevant test cases



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5646) Documentation for scheduling of OPPORTUNISTIC containers


[ 
https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752752#comment-15752752
 ] 

Konstantinos Karanasos commented on YARN-5646:
--

Hi Andrew,

Can you wait until the end of the day for this JIRA?
I am almost done with it and it is the only thing missing for adding the
opportunistic containers feature to alpha-2.

Thanks,
Konstantinos




> Documentation for scheduling of OPPORTUNISTIC containers
> 
>
> Key: YARN-5646
> URL: https://issues.apache.org/jira/browse/YARN-5646
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
>Priority: Blocker
> Attachments: YARN-5646.001.patch
>
>
> This is for adding documentation regarding the scheduling of OPPORTUNISTIC 
> containers.
> It includes both the centralized (YARN-5220) and the distributed (YARN-2877) 
> scheduling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5709) Cleanup leader election configs and pluggability


 [ 
https://issues.apache.org/jira/browse/YARN-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated YARN-5709:
--
Fix Version/s: 3.0.0-alpha2

> Cleanup leader election configs and pluggability
> 
>
> Key: YARN-5709
> URL: https://issues.apache.org/jira/browse/YARN-5709
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Blocker
> Fix For: 3.0.0-alpha2
>
> Attachments: yarn-5709-wip.2.patch, yarn-5709.1.patch, 
> yarn-5709.2.patch, yarn-5709.3.patch, yarn-5709.4.patch
>
>
> While reviewing YARN-5677 and YARN-5694, I noticed we could make the 
> curator-based election code cleaner. It is nicer to get this fixed in 2.8 
> before we ship it, but this can be done at a later time as well. 
> # By EmbeddedElector, we meant it was running as part of the RM daemon. Since 
> the Curator-based elector is also running embedded, I feel the code should be 
> checking for {{!curatorBased}} instead of {{isEmbeddedElector}}
> # {{LeaderElectorService}} should probably be named 
> {{CuratorBasedEmbeddedElectorService}} or some such.
> # The code that initializes the elector should be at the same place 
> irrespective of whether it is curator-based or not. 
> # We seem to be caching the CuratorFramework instance in RM. It makes more 
> sense for it to be in RMContext. If others are okay with it, we might even be 
> better of having {{RMContext#getCurator()}} method to lazily create the 
> curator framework and then cache it. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5646) Documentation for scheduling of OPPORTUNISTIC containers


 [ 
https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated YARN-5646:
--
Target Version/s: 3.0.0-beta1  (was: 3.0.0-alpha2)

> Documentation for scheduling of OPPORTUNISTIC containers
> 
>
> Key: YARN-5646
> URL: https://issues.apache.org/jira/browse/YARN-5646
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
>Priority: Blocker
> Attachments: YARN-5646.001.patch
>
>
> This is for adding documentation regarding the scheduling of OPPORTUNISTIC 
> containers.
> It includes both the centralized (YARN-5220) and the distributed (YARN-2877) 
> scheduling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5464) Server-Side NM Graceful Decommissioning with RM HA


 [ 
https://issues.apache.org/jira/browse/YARN-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated YARN-5464:
--
Target Version/s: 2.9.0, 3.0.0-beta1  (was: 2.9.0, 3.0.0-alpha2)

> Server-Side NM Graceful Decommissioning with RM HA
> --
>
> Key: YARN-5464
> URL: https://issues.apache.org/jira/browse/YARN-5464
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: graceful
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5864) Capacity Scheduler preemption for fragmented cluster


[ 
https://issues.apache.org/jira/browse/YARN-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752695#comment-15752695
 ] 

Wangda Tan commented on YARN-5864:
--

Offline discussed with [~vinodkv].

We can have a better semantic of this feature, which is we can add 
queue-priority property. (Credit to [~vinodkv] for the idea).

In existing scheduler, we sort queues based on (used-capacity / 
configured-capacity). But in some cases we have some apps/services need get 
resource first. For example, we allocate 85% to production queue, and 15% to 
test queue. When production queue is underutilized, we want scheduler give 
resource to production queue first regardless of test queue's utilization.

A rough plan is: we will assign priority to queues under the same parent. Each 
time scheduler picks underutilized queue with highest priority, if there's no 
underutilized queue, scheduler picks queue with lowest utilization.

And when we do preemption, if queue with higher priority has some special 
resource requests, such as very large memory, hard locality, placement 
constraint, etc. Scheduler will do relatively *conservative* preemption from 
other queues with lower priority regardless of utilization. 

That is just a rough idea, [~curino] please let us know your comments. I can 
formalize the design once we can agree with the approach generally.

> Capacity Scheduler preemption for fragmented cluster 
> -
>
> Key: YARN-5864
> URL: https://issues.apache.org/jira/browse/YARN-5864
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-5864.poc-0.patch
>
>
> YARN-4390 added preemption for reserved container. However, we found one case 
> that large container cannot be allocated even if all queues are under their 
> limit.
> For example, we have:
> {code}
> Two queues, a and b, capacity 50:50 
> Two nodes: n1 and n2, each of them have 50 resource 
> Now queue-a uses 10 on n1 and 10 on n2
> queue-b asks for one single container with resource=45. 
> {code} 
> The container could be reserved on any of the host, but no preemption will 
> happen because all queues are under their limits. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2

2016-12-15 Thread Li Lu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752682#comment-15752682
 ] 

Li Lu commented on YARN-4061:
-

I went through the new design doc in HBASE-17018 and I think it's mostly good. 
As we discussed in the weekly sync meeting, one thing we may want to sort out 
here is how to handle the case when the collectors started up and the HBase 
cluster was down. From my point of view, the most conservative approach is to 
assume the HBase cluster was always BAD upon start up. However, the problem is 
we have to spool the very first writes anyways. Can we have a "PROBING" state 
in the coordinator, where we may tolerate slightly longer submission time, to 
let the spooling mutator firstly probe the state of the HBase cluster? Also, 
this probing process may happen before the first write ever comes, so that we 
can do out-of-band probing? 

Another my question is on the idempotent write requirements. Moving my comments 
from google doc to here: 
bq. The spooling mutator itself guarantees an "at least once" semantic? One 
thing I'd like to discuss here is about the write timestamp of each timeline 
writes. I'm not familiar with the HBase code, but are we generating one unique 
timestamp for each write when we actually write them to HBase? If this is the 
case, replaying timeline writes may generate different timestamp and those 
repeated writes may not be idempotent in timeline's perspective?

> [Fault tolerance] Fault tolerant writer for timeline v2
> ---
>
> Key: YARN-4061
> URL: https://issues.apache.org/jira/browse/YARN-4061
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Li Lu
>Assignee: Joep Rottinghuis
>  Labels: YARN-5355
> Attachments: FaulttolerantwriterforTimelinev2.pdf
>
>
> We need to build a timeline writer that can be resistant to backend storage 
> down time and timeline collector failures. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5740) Add a new field in Slider status output - lifetime (remaining)

2016-12-15 Thread Gour Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752645#comment-15752645
 ] 

Gour Saha commented on YARN-5740:
-

[~jianhe] the patch looks good. Can you fix the 6 new checkstyle issues 
including the one below -

h6. SliderClient.java
{code}
if(!appInstances.isEmpty()) {
{code}
add a space after if


> Add a new field in Slider status output - lifetime (remaining)
> --
>
> Key: YARN-5740
> URL: https://issues.apache.org/jira/browse/YARN-5740
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Jian He
> Fix For: yarn-native-services
>
> Attachments: YARN-5740-yarn-native-services.01.patch
>
>
> With YARN-5735, REST service is now setting lifetime to application during 
> submission (YARN-4205 exposed application lifetime support). Now Slider 
> status needs to expose this field so that the REST service can return it in 
> its GET response. Note, the lifetime value that GET response intends to 
> return is the remaining lifetime of the application. 
> There is an ongoing discussion in YARN-4206, that the lifetime value returned 
> in Application Report will be the remaining lifetime (at the time of 
> request). So until it is finalized, the lifetime value might go through 
> different connotations. But as long as we have the lifetime field in the 
> status output, it will be a good start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5600) Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application basis


[ 
https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752626#comment-15752626
 ] 

Miklos Szegedi commented on YARN-5600:
--

I attached a new patch addressing the comments.

> Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application 
> basis
> 
>
> Key: YARN-5600
> URL: https://issues.apache.org/jira/browse/YARN-5600
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Miklos Szegedi
>  Labels: oct16-medium
> Attachments: YARN-5600.000.patch, YARN-5600.001.patch, 
> YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, 
> YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, 
> YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, 
> YARN-5600.011.patch, YARN-5600.012.patch, YARN-5600.013.patch, 
> YARN-5600.014.patch, YARN-5600.015.patch, YARN-5600.016.patch, 
> YARN-5600.017.patch
>
>
> To make debugging application launch failures simpler, I'd like to add a 
> parameter to the CLC to allow an application owner to request delayed 
> deletion of the application's launch artifacts.
> This JIRA solves largely the same problem as YARN-5599, but for cases where 
> ATS is not in use, e.g. branch-2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5600) Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application basis

[
https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752624#comment-15752624
]

Miklos Szegedi commented on YARN-5600:
--

Thank you, [~kasha] for the review!

In general about the design: delete.debug-delay-sec is there for backward
compatibility, I think. I do not expect many people to use it after this new
functionality, since it affects the whole cluster.
Because of this I would not mix it's value much with
delete.max-per-application-debug-delay-sec.
I suggest filing a separate jira to deprecate delete.debug-delay-sec for v3.
This would make the design of this feature simpler.

{quote}
ApplicationConstants: typo in the javadoc.
{quote}
Fixed.
{quote}
YarnConfiguration: max-per-application-debug-delay-sec
"per-application" is redundant and makes the config long. Can we just have
max-debug-delay-sec or d
debug-delay-sec.max.
{quote}
[~vvasudev], You suggested originally this value along with differentiating the
default and the maximum. I am okay with changing it back, what is your opinion?
{quote}
I don't see the need to have a separate constant for the corresponding default
value. Can we use the
value of delete.debug-delay-sec or a multiple of it (say, 2) as default instead?
{quote}
I do not think it is a good idea to set a default to a variable value. I could
not even write anything to yarn-default.xml in this case. Moreover the default
is 0 in 99%+ of the cases.
I would expect people will use the new feature more often in the future, since
it does not affect the whole cluster. delete.debug-delay-sec is there for
backward compatibility. Can we keep it separate than delete.debug-delay-sec?
{quote}
Also, looks like max-debug-delay-sec could be smaller than debug-delay-sec.
That is confusing. Can we make sure max is actually max?
{quote}
I have to defer this to [~vvasudev], since it was his suggestion originally.
{quote}
Move the javadoc from ApplicationImpl to Application. Also, can we linkify
Application.NEVER.
{quote}
I added it to both.
{quote}
If the admin sets debug-delay-sec to a positive value (say, 5 mins) and the
user explicitly sets it to a smaller value (even 0), shouldn't the user's
setting override for all containers of that app?
{quote}
The idea is that the standard value is 0 in 99%+ of cases. Sometimes the user
or the admin wants to delay that by specifying a delay value. Using the max
fulfills both requests, there is no priority, no override. I thought this is
the simplest what I can do to make both of them happy, if I need to specify
both. Limiting the delay would not make much sense, since any smaller number
what requested prevents them to collect the necessary debug information for the
other party. There is a feature to disable the delay functionality in sensitive
applications and that is the cluster wide max delay.
{quote}
It appears we allow setting this value on a per-container basis through
ContainerLaunchContext, but consider only the largest value. Is there value in
allowing specifying this on a per-container basis. Why not specify it on a
per-app basis? Say, through ApplicationSubmissionContext?
{quote}
We consider all values specified in each CLC. However, if an application has
multiple containers on the same node, we delete the application dir only after
the latest CLC is deleted. The largest logic just makes sure that the
application dir is not deleted before the containers in it are deleted. Is not
it more flexible to specify a container specific value?
{quote}
Any specific reason for capturing the deletion time in Date instead of long?
Isn't Date more expensive?
{quote}
Date is just a wrapper above a long value. I feel the code is more readable
with Date than it would be otherwise. You can for example get the real time
value in the debugger without any additional calculations.
{quote}
yarn-default: s/mumber/number
{quote}
Fixed.
{quote}
DeletionServe
scheduleFileDeletionTask takes a delay, and then adjusts it based on
configuration. These seems error-prone. Callers of this method would assume the
file will be deleted at the provided time. Can the caller take care of the
adjustment?
{quote}
It depends. If this is true, it was already error prone before my change, since
we requested a delete that happened later overridden by the default delay. All
I do is adding a custom delay. The contract is that the task will be delayed
for at least the amount of time specified. I was thinking about changing the
function name or creating a separate function that reflects this design better
that the caller would call. Honestly, scheduleFileDeletionTask should be
refactored anyways to be private, so I just changed the documentation for now.
What do you think? (This problem would also go away, if we deprecate default
delay.)
{quote}
getEffectiveDelay: if the limitedDelay > maxDebugDelay, one could set
limitedDelay to maxDebugDelay

[jira] [Updated] (YARN-5600) Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application basis


 [ 
https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Miklos Szegedi updated YARN-5600:
-
Attachment: YARN-5600.017.patch

> Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application 
> basis
> 
>
> Key: YARN-5600
> URL: https://issues.apache.org/jira/browse/YARN-5600
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Miklos Szegedi
>  Labels: oct16-medium
> Attachments: YARN-5600.000.patch, YARN-5600.001.patch, 
> YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, 
> YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, 
> YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, 
> YARN-5600.011.patch, YARN-5600.012.patch, YARN-5600.013.patch, 
> YARN-5600.014.patch, YARN-5600.015.patch, YARN-5600.016.patch, 
> YARN-5600.017.patch
>
>
> To make debugging application launch failures simpler, I'd like to add a 
> parameter to the CLC to allow an application owner to request delayed 
> deletion of the application's launch artifacts.
> This JIRA solves largely the same problem as YARN-5599, but for cases where 
> ATS is not in use, e.g. branch-2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5216) Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM


[ 
https://issues.apache.org/jira/browse/YARN-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752545#comment-15752545
 ] 

Hadoop QA commented on YARN-5216:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m 
47s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
56s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
40s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
45s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
54s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
14s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
21s{color} | {color:green} YARN-5972 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 51s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 1 new + 207 unchanged - 0 fixed = 208 total (was 207) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
30s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
26s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 
57s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
29s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 35s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5216 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843350/YARN-5216-YARN-5972.004.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux ef1a03dcff13 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-5972 / 8752f53 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |

[jira] [Updated] (YARN-5962) Spelling errors in logging and exceptions for resource manager code

2016-12-15 Thread Grant Sohn (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Sohn updated YARN-5962:
-
Attachment: YARN-5962.3.patch

Fixed style changes.

> Spelling errors in logging and exceptions for resource manager code
> ---
>
> Key: YARN-5962
> URL: https://issues.apache.org/jira/browse/YARN-5962
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Grant Sohn
>Assignee: Grant Sohn
>Priority: Trivial
> Attachments: YARN-5962.1.patch, YARN-5962.2.patch, YARN-5962.3.patch
>
>
> Found spelling errors in exceptions and logging.
> Examples:
> Invailid -> Invalid
> refinition -> definition
> non-exsisting -> non-existing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Resolved] (YARN-5882) Test only changes from YARN-4126


 [ 
https://issues.apache.org/jira/browse/YARN-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved YARN-5882.
---
   Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
   2.8.0

Resolving this JIRA to log that the initial commit for YARN-4126 and subsequent 
addendum commit left the test-only changes in branch-2, branch-2.8, and trunk. 
We like these changes, but it needs a resolved JIRA to show up in the release 
notes.

For future reference, the two commits are as follows:

{noformat}
commit ada876cd1d22b61f237603cf339bbed65285dab8
Author: Jian He 
Date:   Tue Dec 13 20:49:54 2016 -0800

Revert YARN-4126. RM should not issue delegation tokens in unsecure mode.

commit e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7
Author: Jian He 
Date:   Mon Sep 14 14:09:19 2015 +0800

YARN-4126. RM should not issue delegation tokens in unsecure mode. 
Contributed by Bibin A Chundatt
{noformat}

> Test only changes from YARN-4126
> 
>
> Key: YARN-5882
> URL: https://issues.apache.org/jira/browse/YARN-5882
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Jian He
> Fix For: 2.8.0, 3.0.0-alpha2
>
>
> Filed this for tracking. We might want to revert YARN-4126 based on 
> discussion in that JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5882) Test only changes from YARN-4126


 [ 
https://issues.apache.org/jira/browse/YARN-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated YARN-5882:
--
  Assignee: Jian He
  Priority: Major  (was: Blocker)
Issue Type: Improvement  (was: Bug)
   Summary: Test only changes from YARN-4126  (was: Reconsider YARN-4126 RM 
should not issue delegation tokens in unsecure mode)

> Test only changes from YARN-4126
> 
>
> Key: YARN-5882
> URL: https://issues.apache.org/jira/browse/YARN-5882
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Jian He
>
> Filed this for tracking. We might want to revert YARN-4126 based on 
> discussion in that JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4126) RM should not issue delegation tokens in unsecure mode


[ 
https://issues.apache.org/jira/browse/YARN-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752344#comment-15752344
 ] 

Andrew Wang commented on YARN-4126:
---

Thanks for reverting [~jianhe], though in the future anything that's not a full 
revert should go through code review, and not be committed with a message that 
says "Revert ...".

I'll use YARN-5882 to log the fact that essentially test-only changes that were 
made, to trunk/branch-2/branch-2.8. Otherwise the test changes aren't logged by 
any JIRA.

> RM should not issue delegation tokens in unsecure mode
> --
>
> Key: YARN-4126
> URL: https://issues.apache.org/jira/browse/YARN-4126
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Jian He
>Assignee: Bibin A Chundatt
> Fix For: 3.0.0-alpha1
>
> Attachments: 0001-YARN-4126.patch, 0002-YARN-4126.patch, 
> 0003-YARN-4126.patch, 0004-YARN-4126.patch, 0005-YARN-4126.patch, 
> 0006-YARN-4126.patch
>
>
> ClientRMService#getDelegationToken is currently  returning a delegation token 
> in insecure mode. We should not return the token if it's in insecure mode. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5993) Allow native services quicklinks to be exported for each component


[ 
https://issues.apache.org/jira/browse/YARN-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752132#comment-15752132
 ] 

Hadoop QA commented on YARN-5993:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
39s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
31s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
34s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m  
9s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
 in yarn-native-services has 262 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
32s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services 
failed. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 18s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core:
 The patch generated 6 new + 169 unchanged - 7 fixed = 175 total (was 176) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
16s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
 generated 1 new + 261 unchanged - 1 fixed = 262 total (was 262) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
31s{color} | {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
51s{color} | {color:green} hadoop-yarn-slider-core in the patch passed. {color} 
|
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 53s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
 |
|  |  
org.apache.slider.providers.docker.DockerProviderService.publishExportGroups(String,
 String, String, String, List) makes inefficient use of keySet iterator instead 
of entrySet iterator  At DockerProviderService.java:makes inefficient use of 
keySet iterator instead of entrySet iterator  At 
DockerProviderService.java:[line 368] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5993 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843445/YARN-5993-yarn-native-services.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 57b2824b4aef 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64

[jira] [Updated] (YARN-5993) Allow native services quicklinks to be exported for each component

2016-12-15 Thread Billie Rinaldi (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billie Rinaldi updated YARN-5993:
-
Attachment: YARN-5993-yarn-native-services.001.patch

> Allow native services quicklinks to be exported for each component
> --
>
> Key: YARN-5993
> URL: https://issues.apache.org/jira/browse/YARN-5993
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Attachments: YARN-5993-yarn-native-services.001.patch
>
>
> The quicklinks export capability changed in switching from the agent provider 
> to the docker provider, and currently the docker provider only allows one 
> component to export links. We should improve this capability to be more in 
> line with the previous agent capability.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6007) Unit test failures in Yarn due to java.lang.NoClassDefFoundError

2016-12-15 Thread Haibo Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751905#comment-15751905
 ] 

Haibo Chen commented on YARN-6007:
--

Thanks for reporting this [~Sonia]. YARN-5976 is currently in flight to upgrade 
HBASE to 1.2.4. 

> Unit test failures in Yarn due to java.lang.NoClassDefFoundError
> 
>
> Key: YARN-6007
> URL: https://issues.apache.org/jira/browse/YARN-6007
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha2
> Environment: $ uname -a
> Linux f8eef0f055cf 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 
> 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Sonia Garudi
>  Labels: ppc64
> Attachments: YARN-6007.patch
>
>
> In Hadoop Yarn project, 21 unit test failures are seen with below error :
> java.lang.NoClassDefFoundError: Could not initialize class 
> org.apache.hadoop.hbase.util.Bytes$LexicographicalComparerHolder$UnsafeComparer
> java.lang.NullPointerException: null
>  at org.apache.hadoop.yarn.server.timelineservice.storage.
> TestHBaseTimelineStorageApps.tearDownAfterClass(TestHBaseTimelineStorageApps.java:1847)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5650) Render Application Timeout value in web UI(new and old)

2016-12-15 Thread Sunil G (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751896#comment-15751896
 ] 

Sunil G commented on YARN-5650:
---

javadoc issue reported on AppBlock is known and its done as per hamlet 
framework. Test not needed as its UI.

Patch generally looks fine for me. I will wait for comments from 
[~rohithsharma] if any.

> Render Application Timeout value in web UI(new and old)
> ---
>
> Key: YARN-5650
> URL: https://issues.apache.org/jira/browse/YARN-5650
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
> Attachments: YARN-5650.001.patch, YARN-5650.002.patch, 
> YARN-5650.003.patch, YARN-5650.004.patch, YARN-5650.005.patch, 
> screenshot-1.png, screenshot-2.png
>
>
> This is to track timeout rendering in old and new yarn web UI



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6006) Log aggregation causes nodemanager OOM

2016-12-15 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751885#comment-15751885
 ] 

Naganarasimha G R commented on YARN-6006:
-

Hi [~shenyinjie],
Agree that during the writeVersion it might have collected OOM but it might not 
 be the exact reason for the OOM. Did you get to take heap/memory dump ? 

> Log aggregation causes nodemanager OOM
> --
>
> Key: YARN-6006
> URL: https://issues.apache.org/jira/browse/YARN-6006
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Shen Yinjie
> Attachments: YARN-6006_1.patch, e.png
>
>
> log aggregation is enabled, nodemanager died with oom exception. exception as 
> sreenshot. and NM_heapsize is 4096MB



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5877) Allow all nm-whitelist-env to get overridden during launch


[ 
https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751858#comment-15751858
 ] 

Hadoop QA commented on YARN-5877:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 
32s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 11s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5877 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843435/YARN-5877.0006.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 40746e4fe737 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 36947f7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14330/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14330/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Allow all nm-whitelist-env to get overridden during launch
> --
>
> Key: YARN-5877
> URL: https://issues.apache.org/jira/browse/YARN-5877
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: Dockerfile, YARN-5877.0001.patch, YARN-5877.0002.patch, 
> YARN-5877.0003.patch, YARN-5877.0004.patch, YARN-5877.0005.patch, 
>

[jira] [Commented] (YARN-3409) Add constraint node labels

2016-12-15 Thread Naganarasimha G R (JIRA)

[
https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751851#comment-15751851
]

Naganarasimha G R commented on YARN-3409:
-

Thanks [~wangda], [~kkaranasos] & [~grey], for sharing your thoughts.
I agree the general idea of Top down(coming from the requirements) but lets not
again limit ourself to current problem and bring up the API and later have
minimal bandwidth for future requirements (due to compatability issues). And
based on past experience it will take long time to come back and update once we
define the interface. Besides we do have some scenarios where in we require
numerical comparisons are required at the minimum.

I kind of agree on some of @wangda's points like exposing {{public
constraint-type}} would be added overhead to maintain it as compatabile
interface even in future. but the point i would want to make is atleast we
should have predetermined types (like wangda mentioned earlier String, Long,
Version (long array followed by a string)). But i would still prefer to have a
constraint type (defaulting to string) so that the request doesnt hang because
of wrong expression being provided or wrongly select as particular type based
on regex. If its predefined types then its easier to inform as part of response
or throw an exception that the given expresion is invalid. it would definitely
save the processing time too, when they do not match.
For other complexitites mentioned :
{quote}
* Change type of constraint
* Remove/Add constraint (Remove constraint need to check if it is associate
with other node, etc.)
{quote}
This would be same as modifying the existing partition type (exclusive to non
exclusive and vice versa). And i feels its not too complex and as you mentioned
if the number of references to this constraint becomes zero then we can allow
to remove it. Similar thing is already supported for partitions currently

Going one by one on the examples taken would be a long debatable discussion but
at the least one straight forward scenario for numerical comparison which we
have is number of disks, We have CarbonData(incubator apache project) which is
kind of more of OLAP processing. We usually deploy with other bigdata projects
and hence its more like heterogenous workloads on hetergenous hardware. When we
do bulk load of data(TB's) into CarbonData (using spark) we tend to have lot of
intermediate data, To parallelize the processing we not only require more cores
but good number of disks too, based on the partitions we generally could
determine how many disks would be good to achieve better load performance. If
we bring in the number of disks as resource type it will be enforced on all the
containers running in the cluster and it will limit the cluster usability. So
it would be ideal to specify at least for these kind of tasks to get the Nodes
with at least the specified number of disks to improve the load performance
with additional constraints like anti affinity to aid it.
Can explain (/rather debate :) ) other scenarios too, may be in the offline
meeting.

bq. For example, using simple labels, one node can be annotated with label
"Java6". Then a task that requires at least Java 5 can request for a node with
"Java5 || Java6". I think that with our current use cases, this will be
sufficient.
These kind of Version comparisions will have impact when admin upgrades the
Java to particular higher version. Then apps which are already running will be
impacted. Admin will not be aware of the apps which are dependent on the Java
version. So good thing would be to have version > x.y.z kinds

bq. Could you add some references here?
Well may be i dint state what i intended or it was wrongly took as other
scheduler support, Different schedulers like IBM LSF,Mesos and Kubernetes
support labels different in different ways for different scenarios, Some
support expressions on load average/swap, some support different kind of
expressions(SET, RANGE) and some support both string based constraints. So i
tried to generalize it as extendable customized labels and not intended to say
that its already been supported.

LSF :
http://www.ibm.com/support/knowledgecenter/SSETD4_9.1.3/lsf_admin/select_string.html
MESOS : http://mesos.apache.org/documentation/latest/attributes-resources/
KUBERNETES : http://kubernetes.io/docs/user-guide/node-selection/

Regarding concidering antiaffinity and affinity within the scope of
constraints, i had few queries/concerns
# current syntax what you mentioned captures for antiaffinity and affinity with
apps, but how about app's priority(different type of tasks) ? looks very
specific syntax needs to be introduced for that too ?
# According to YARN-1042 you had hinted out target as host/rack/partition. If
we start supporting within constraints i doubt we will be later consider these
kind of requirements in future, or

[jira] [Updated] (YARN-5877) Allow all nm-whitelist-env to get overridden during launch

2016-12-15 Thread Bibin A Chundatt (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bibin A Chundatt updated YARN-5877:
---
Attachment: YARN-5877.0006.patch

> Allow all nm-whitelist-env to get overridden during launch
> --
>
> Key: YARN-5877
> URL: https://issues.apache.org/jira/browse/YARN-5877
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: Dockerfile, YARN-5877.0001.patch, YARN-5877.0002.patch, 
> YARN-5877.0003.patch, YARN-5877.0004.patch, YARN-5877.0005.patch, 
> YARN-5877.0006.patch, bootstrap.sh, yarn-site.xml
>
>
> As per the {{yarn.nodemanager.env-whitelist}} for the configured values 
> should  containers may override rather than use NodeManager's default.
> {code}
>   
> Environment variables that containers may override rather 
> than use NodeManager's default.
> yarn.nodemanager.env-whitelist
> 
> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME
>   
> {code}
> But only the following containers can override
> {code}
> whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
> whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5650) Render Application Timeout value in web UI(new and old)


[ 
https://issues.apache.org/jira/browse/YARN-5650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751380#comment-15751380
 ] 

Hadoop QA commented on YARN-5650:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
48s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 44s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 1 new + 79 unchanged - 1 fixed = 80 total (was 80) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
16s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common 
generated 2 new + 159 unchanged - 0 fixed = 161 total (was 159) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
28s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 42m 
38s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
28s{color} | {color:green} hadoop-yarn-ui in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
30s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m  2s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5650 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843402/YARN-5650.005.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs

[jira] [Commented] (YARN-5967) Fix slider core module findbugs warnings


[ 
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751228#comment-15751228
 ] 

Hadoop QA commented on YARN-5967:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
 7s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
53s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
58s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  3m 
22s{color} | {color:red} hadoop-yarn in yarn-native-services failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
22s{color} | {color:green} yarn-native-services passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
58s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
 in yarn-native-services has 262 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  1m 
44s{color} | {color:red} hadoop-yarn in yarn-native-services failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
32s{color} | {color:red} hadoop-yarn-slider in yarn-native-services failed. 
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
31s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services 
failed. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 57s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 27 new + 842 unchanged - 55 fixed = 869 total (was 897) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  3m 
13s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider 
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
4s{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
 generated 0 new + 27 unchanged - 235 fixed = 27 total (was 262) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  1m 
46s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
31s{color} | {color:red} hadoop-yarn-slider in the patch failed. {color} |
|

[jira] [Commented] (YARN-3053) [Security] Review and implement security in ATS v.2

2016-12-15 Thread Rohith Sharma K S (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751222#comment-15751222
 ] 

Rohith Sharma K S commented on YARN-3053:
-

Apologies for pitching very late. A quick question,  does this JIRA focusing on 
both authentication and authorization or only authentication? I see document is 
mainly focusing on authentication. 

> [Security] Review and implement security in ATS v.2
> ---
>
> Key: YARN-3053
> URL: https://issues.apache.org/jira/browse/YARN-3053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>  Labels: YARN-5355
> Attachments: ATSv2Authentication(draft).pdf
>
>
> Per design in YARN-2928, we want to evaluate and review the system for 
> security, and ensure proper security in the system.
> This includes proper authentication, token management, access control, and 
> any other relevant security aspects.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5650) Render Application Timeout value in web UI(new and old)

2016-12-15 Thread Akhil PB (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akhil PB updated YARN-5650:
---
Attachment: YARN-5650.005.patch

> Render Application Timeout value in web UI(new and old)
> ---
>
> Key: YARN-5650
> URL: https://issues.apache.org/jira/browse/YARN-5650
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
> Attachments: YARN-5650.001.patch, YARN-5650.002.patch, 
> YARN-5650.003.patch, YARN-5650.004.patch, YARN-5650.005.patch, 
> screenshot-1.png, screenshot-2.png
>
>
> This is to track timeout rendering in old and new yarn web UI



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


[ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750769#comment-15750769
 ] 

zhangyubiao edited comment on YARN-5995 at 12/15/16 12:00 PM:
--

YARN-5995-v1.patch is commit 


was (Author: piaoyu zhang):
YARN-59950-v1.patch is commit 

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Fix For: 2.7.4, 3.0.0-beta1
>
> Attachments: YARN-5995-v1.patch, YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6006) Log aggregation causes nodemanager OOM


[ 
https://issues.apache.org/jira/browse/YARN-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751107#comment-15751107
 ] 

Hadoop QA commented on YARN-6006:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
25s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 21s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6006 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843369/YARN-6006_1.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux ffd6dea65476 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 64a2d5b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14327/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14327/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Log aggregation causes nodemanager OOM
> --
>
> Key: YARN-6006
> URL: https://issues.apache.org/jira/browse/YARN-6006
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Shen Yinjie
> Attachments: YARN-6006_1.patch, e.png
>
>
> log aggregation is enabled, nodemanager died with oom exception. exception as 
>

[jira] [Commented] (YARN-5939) FSDownload leaks FileSystem resources


[ 
https://issues.apache.org/jira/browse/YARN-5939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751103#comment-15751103
 ] 

Hadoop QA commented on YARN-5939:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: 
The patch generated 0 new + 101 unchanged - 1 fixed = 101 total (was 102) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
19s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5939 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843374/YARN-5939.03.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux bc72195af64b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 64a2d5b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/14325/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/14325/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> FSDownload leaks FileSystem resources
> -
>
> Key: YARN-5939
> URL: https://issues.apache.org/jira/browse/YARN-5939
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.5.1, 2.7.3
>Reporter: liuxiangwei
>Assignee: Weiwei Yang
> Attachments: YARN-5939.01.patch, YARN-5939.02.patch, 
> YARN-5939.03.patch
>
>   Original Estimate:

[jira] [Updated] (YARN-6007) Unit test failures in Yarn due to java.lang.NoClassDefFoundError

2016-12-15 Thread Sonia Garudi (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sonia Garudi updated YARN-6007:
---
Attachment: YARN-6007.patch

> Unit test failures in Yarn due to java.lang.NoClassDefFoundError
> 
>
> Key: YARN-6007
> URL: https://issues.apache.org/jira/browse/YARN-6007
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha2
> Environment: $ uname -a
> Linux f8eef0f055cf 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 
> 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Sonia Garudi
>  Labels: ppc64
> Attachments: YARN-6007.patch
>
>
> In Hadoop Yarn project, 21 unit test failures are seen with below error :
> java.lang.NoClassDefFoundError: Could not initialize class 
> org.apache.hadoop.hbase.util.Bytes$LexicographicalComparerHolder$UnsafeComparer
> java.lang.NullPointerException: null
>  at org.apache.hadoop.yarn.server.timelineservice.storage.
> TestHBaseTimelineStorageApps.tearDownAfterClass(TestHBaseTimelineStorageApps.java:1847)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6007) Unit test failures in Yarn due to java.lang.NoClassDefFoundError

2016-12-15 Thread Sonia Garudi (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751045#comment-15751045
 ] 

Sonia Garudi commented on YARN-6007:


On debugging, we found the issue is due to Hadoop's dependency on Hbase version 
1.1.3. The errors are related to HBASE-15322. The fix for the given error is in 
version 1.1.4.

On upgrading Hbase version to 1.1.4, the build fails with convergence errors. 
On adding few exclusions in the pom.xml file, the build is successful, and the 
unit test errors are resolved. I have attached a patch with the changes made.

> Unit test failures in Yarn due to java.lang.NoClassDefFoundError
> 
>
> Key: YARN-6007
> URL: https://issues.apache.org/jira/browse/YARN-6007
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha2
> Environment: $ uname -a
> Linux f8eef0f055cf 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 
> 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Sonia Garudi
>  Labels: ppc64
>
> In Hadoop Yarn project, 21 unit test failures are seen with below error :
> java.lang.NoClassDefFoundError: Could not initialize class 
> org.apache.hadoop.hbase.util.Bytes$LexicographicalComparerHolder$UnsafeComparer
> java.lang.NullPointerException: null
>  at org.apache.hadoop.yarn.server.timelineservice.storage.
> TestHBaseTimelineStorageApps.tearDownAfterClass(TestHBaseTimelineStorageApps.java:1847)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6007) Unit test failures in Yarn due to java.lang.NoClassDefFoundError

2016-12-15 Thread Sonia Garudi (JIRA)

Sonia Garudi created YARN-6007:
--

 Summary: Unit test failures in Yarn due to 
java.lang.NoClassDefFoundError
 Key: YARN-6007
 URL: https://issues.apache.org/jira/browse/YARN-6007
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn
Affects Versions: 3.0.0-alpha2
 Environment: $ uname -a
Linux f8eef0f055cf 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:42:36 
UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
Reporter: Sonia Garudi


In Hadoop Yarn project, 21 unit test failures are seen with below error :

java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.hadoop.hbase.util.Bytes$LexicographicalComparerHolder$UnsafeComparer
java.lang.NullPointerException: null
 at org.apache.hadoop.yarn.server.timelineservice.storage.
TestHBaseTimelineStorageApps.tearDownAfterClass(TestHBaseTimelineStorageApps.java:1847)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5967) Fix slider core module findbugs warnings


 [ 
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5967:
--
Attachment: YARN-5967-yarn-native-services.05.patch

> Fix slider core module findbugs warnings 
> -
>
> Key: YARN-5967
> URL: https://issues.apache.org/jira/browse/YARN-5967
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5967-yarn-native-services.01.patch, 
> YARN-5967-yarn-native-services.02.patch, 
> YARN-5967-yarn-native-services.02.patch, 
> YARN-5967-yarn-native-services.03.patch, 
> YARN-5967-yarn-native-services.04.patch, 
> YARN-5967-yarn-native-services.04.patch, 
> YARN-5967-yarn-native-services.05.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5967) Fix slider core module findbugs warnings


 [ 
https://issues.apache.org/jira/browse/YARN-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5967:
--
Attachment: YARN-5967-yarn-native-services.04.patch

> Fix slider core module findbugs warnings 
> -
>
> Key: YARN-5967
> URL: https://issues.apache.org/jira/browse/YARN-5967
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5967-yarn-native-services.01.patch, 
> YARN-5967-yarn-native-services.02.patch, 
> YARN-5967-yarn-native-services.02.patch, 
> YARN-5967-yarn-native-services.03.patch, 
> YARN-5967-yarn-native-services.04.patch, 
> YARN-5967-yarn-native-services.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5939) FSDownload leaks FileSystem resources

2016-12-15 Thread Weiwei Yang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-5939:
--
Attachment: YARN-5939.03.patch

> FSDownload leaks FileSystem resources
> -
>
> Key: YARN-5939
> URL: https://issues.apache.org/jira/browse/YARN-5939
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.5.1, 2.7.3
>Reporter: liuxiangwei
>Assignee: Weiwei Yang
> Attachments: YARN-5939.01.patch, YARN-5939.02.patch, 
> YARN-5939.03.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Background
> To use our self-defined FileSystem class, the item of configuration 
> "fs.%s.impl.disable.cache" should set to true.
> In YARN's source code, the class named 
> "org.apache.hadoop.yarn.util.FSDownload" use getFileSystem but never close, 
> which leading to file descriptor leak because our self-defined FileSystem 
> class close the file descriptor when the close function is invoked.
> My Question below:
> 1. whether invoking "getFileSystem" but never close is YARN's expected 
> behavior 
> 2. what should we do in our self-defined FileSystem resolve it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Reopened] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


 [ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangyubiao reopened YARN-5995:
---

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Fix For: 2.7.4, 3.0.0-beta1
>
> Attachments: YARN-5995-v1.patch, YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


 [ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangyubiao updated YARN-5995:
--
Fix Version/s: (was: 2.7.1)
   2.7.4

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Fix For: 2.7.4, 3.0.0-beta1
>
> Attachments: YARN-5995-v1.patch, YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


 [ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangyubiao updated YARN-5995:
--
Target Version/s: 3.0.0-beta1
   Fix Version/s: 2.7.1
  3.0.0-beta1

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Fix For: 2.7.4, 3.0.0-beta1
>
> Attachments: YARN-5995-v1.patch, YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


[ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750769#comment-15750769
 ] 

zhangyubiao commented on YARN-5995:
---

YARN-59950-v1.patch is commit 

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Attachments: YARN-5995-v1.patch, YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5995) Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition performance


 [ 
https://issues.apache.org/jira/browse/YARN-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangyubiao updated YARN-5995:
--
Attachment: YARN-5995-v1.patch

> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance
> ---
>
> Key: YARN-5995
> URL: https://issues.apache.org/jira/browse/YARN-5995
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: metrics, resourcemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.2 Hadoop-2.7.1 
>Reporter: zhangyubiao
>  Labels: patch
> Attachments: YARN-5995-v1.patch, YARN-5995.patch
>
>
> Add RMStateStore metrics to monitor all RMStateStoreEventTypeTransition 
> performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6006) Log aggregation causes nodemanager OOM


 [ 
https://issues.apache.org/jira/browse/YARN-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shen Yinjie updated YARN-6006:
--
Description: log aggregation is enabled, nodemanager died with oom 
exception. exception as sreenshot. and NM_heapsize is 4096MB  (was: log 
aggregation is enabled, nodemanager died with oom exception. exception as 
sreenshot.)

> Log aggregation causes nodemanager OOM
> --
>
> Key: YARN-6006
> URL: https://issues.apache.org/jira/browse/YARN-6006
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Shen Yinjie
> Attachments: YARN-6006_1.patch, e.png
>
>
> log aggregation is enabled, nodemanager died with oom exception. exception as 
> sreenshot. and NM_heapsize is 4096MB



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-6006) Log aggregation causes nodemanager OOM


[ 
https://issues.apache.org/jira/browse/YARN-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750717#comment-15750717
 ] 

Shen Yinjie edited comment on YARN-6006 at 12/15/16 8:11 AM:
-

refer to the code in {{AggregatedLogFormat.java}}
{code}private void writeVersion() throws IOException {
  DataOutputStream out = this.writer.prepareAppendKey(-1);
  VERSION_KEY.write(out);
  out.close();
  out = this.writer.prepareAppendValue(-1);
  out.writeInt(VERSION);
  out.close();
} {code}
{{DataOutputStream out}} may cannot be closed while exception occurs.so need to 
add {{finally}} module to ensure stream  closed..


was (Author: shenyinjie):
refer to code.
{code}private void writeVersion() throws IOException {
  DataOutputStream out = this.writer.prepareAppendKey(-1);
  VERSION_KEY.write(out);
  out.close();
  out = this.writer.prepareAppendValue(-1);
  out.writeInt(VERSION);
  out.close();
} {code}
{{DataOutputStream out}} may cannot be closed while exception occurs.so need to 
add {{finally}} module to ensure stream  closed..

> Log aggregation causes nodemanager OOM
> --
>
> Key: YARN-6006
> URL: https://issues.apache.org/jira/browse/YARN-6006
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Shen Yinjie
> Attachments: YARN-6006_1.patch, e.png
>
>
> log aggregation is enabled, nodemanager died with oom exception. exception as 
> sreenshot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6006) Log aggregation causes nodemanager OOM


 [ 
https://issues.apache.org/jira/browse/YARN-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shen Yinjie updated YARN-6006:
--
Attachment: YARN-6006_1.patch

attach patch fixed that.

> Log aggregation causes nodemanager OOM
> --
>
> Key: YARN-6006
> URL: https://issues.apache.org/jira/browse/YARN-6006
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Shen Yinjie
> Attachments: YARN-6006_1.patch, e.png
>
>
> log aggregation is enabled, nodemanager died with oom exception. exception as 
> sreenshot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6006) Log aggregation causes nodemanager OOM